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Chapter I 



Introduction 



I n 2006 and 2007, the U.S. Department of Education (ED) awarded $478 million in 
grants from the Teacher Incentive Fund (TIF) to support the development of innovative 
teacher compensation strategies. New Leaders for New Schools (NLNS), together with 
five partners — Memphis City Schools (MCS), the District of Columbia Public Schools 
(DCPS), Denver Public Schools, Prince Georges County Public Schools, and a consortium 
of charter schools — have received more than $88 million of these funds to implement the 
Effective Practice Incentive Community (EPIC) program. EPIC offers performance-based 
awards to staff in high-performing schools in return for their agreement to cooperate with 
New Leaders’ efforts to document and share effective teaching practices. 

Mathematica Policy Research, Inc., (MPR) has been hired to evaluate the EPIC initiative 
in Memphis, DC, and the charter consortium. 1 In this report, we present our evaluation 
design. We detail our estimation procedures, document what is currently known about how 
EPIC operates within the various partners, and provide a workplan for the evaluation 
moving forward. In this chapter, we provide an overview of EPIC and present our general 
approach to conducting the evaluation. Chapter II presents our approach to evaluating EPIC 
in Memphis. Chapter III includes our approach to evaluating EPIC in Washington, DC. 
Chapter IV presents our approach to evaluating EPIC for the charter school consortium." 



Overview of Epic 

EPIC has two key features. First, NLNS offers performance-based awards to staff in 
high-performing schools, where high performance is identified based on student 
achievement gains. The awards are made to principals, teachers, and in some cases, 
additional staff. The awards are designed primarily to encourage staff from high-performing 



1 We do not present evaluation designs for Denver or Prince George’s County in this report. MPR is not 
conducting the evaluation of EPIC in Denver and has not yet started work with Prince George’s County. 

2 This report’s discussion of EPIC implementation and programmatic details reflects MPR’s 
understanding as of March 2008, when this report was written. 
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schools to help NLNS document effective practices. In addition, eligibility for these awards 
may operate as an incentive for principals and teachers to improve student achievement. 

Second, EPIC includes an intensive effort to document and disseminate effective 
practices. NLNS works with the highest performing teachers in the highest performing 
schools to identify practices that may contribute to the observed growth in student 
achievement. These practices are then disseminated to all schools within the partner 
organizations, as well as to all schools across the nation. The primary motivation for EPIC is 
the belief that these dissemination activities will result in changes in teacher practices and 
thereby cause widespread improvements in student achievement. 

Our evaluation of EPIC will examine both the incentive awards and the effective 
practice components. 

Incentive Awards 

NLNS is making financial awards to staff in high-performing schools in the 2007/2008 
through 2011/2012 school years. The payments are supplemental income for the staff to be 
used as they see fit. 

In the 2007/2008 school year, NLNS made two types of EPIC Awards: 

• Gold-Gain awards were made to the highest-performing schools of each 
partner; and 

• Silver-Gain awards were made to the second tier of high-performing schools 
of each partner. 

Eligibility criteria for an EPIC incentive award vary from partner to partner. In 
Memphis, only schools with 50 percent or more of the population eligible for a free or 
reduced price lunch (F/RPL) are eligible. In DC, all traditional schools are required to 
participate in the incentive program; however, TIF funds are only used to finance awards for 
schools with at least 30 percent of students eligible for F/RPL, while district or private funds 
finance the remainder of the awards. In the charter school consortium, only schools widi 30 
percent or more of the student population eligible for F/RPL were recmited. Additional 
factors can affect a school’s eligibility for EPIC incentive awards, and those will be covered 
in subsequent pages. 

The criteria used to identify Gold-Gain and Silver-Gain schools also differed from 
partner to partner. In Memphis and among the charter schools, awards were given to 
schools with the largest estimates of “value added” to the achievement of their students, as 
measured by a statistical model that estimates the achievement gains of individual students, 
adjusting for student background characteristics. Gold-Gain schools were those with the 
largest value added model (YAM) scores, and Silver-Gain schools were in the next rank in 
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terms of VAM scores. 3 In DC only Gold-Gain schools were awarded, and they were the 
schools with the largest increases in the percent of students that were proficient on state 
tests. In later years DC also plans to use a VAM model to identify award-winning schools. 

The number of schools that received Gold-Gain and Silver-Gain awards varied from 
partner to partner (Table 1.1). The number of awardees was determined by NLNS based in 
part on the distribution of high-performing schools. Other district-specific criteria were used 
to determine the number of schools in each category. 



Table 1.1. Number of EPIC Award-Winning Schools in 2007/2008 





Eligible Schools 


Schools Receiving 
Gold-Gain Awards 


Schools Receiving 
Silver-Gain 
Awards 


Memphis 


136 


5 


12 


DC a 


132 


3 


n.a. 


Charter School Consortium 


97 


7 


15 



a DC’s EPIC program — called TEAM — does not include Silver-Gain schools. 

A central goal of the 2007/2008 awards was to make teachers aware of the EPIC 
incentive program. For these early awards, there was no incentive period in which teachers 
could alter behavior in an effort to receive the award. This is because these awards were 
given out based on performance during the 2006/2007 school year, a period when school 
staff were unaware of EPIC. Awards to be made in 2008/2009 will be based on performance 
during the 2007/2008 school year when staff are aware of the program so these awards (and 
awards given in subsequent years) may serve as an incentive for school staff to increase 
student achievement. 

The types of staff who received an award as well as the size of the award also differed 
from partner to partner. Principals, assistant principals and instructional staff received 
awards in Gold-Gain and Silver-Gain schools in all three partners. In DC, other building 
staff (guidance counselors and school support personnel, such as custodial and lunch service 
staff) also received awards. The size of the award payments varied substantially. For 
example, in 2007/2008, principals at Gold-Gain schools received between $10,000 and 
$20,000, and instructional staff in Gold-Gain schools received between $1,500 and $8,000 
(Table 1.2). 



3 VAM scores estimate the average contribution that each school makes to changes in student 
achievement from one year to another, holding constant other factors that influence student achievement. See 
Booker and Isenberg (2008) and Booker et al. (2008) for details on how MPR estimated VAM models for 
Memphis and Charter schools respectively based on 2006-2007 performance. 
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Table 1.2. EPIC Incentive Award Amounts in 2007/2008 



Silver-Gain 





Gold-Gain Schools 


Schools 


Memphis 


Principals 


$15,000 


$10,000 


Assistant principals 


10,000 


7,500 


All instructional staff 


1,500 


1,000 


DC a 


Principals 


10,000 


n.a. 


Assistant principals 


9,000 


n.a. 


All instructional staff 


8,000 


n.a. 


Guidance counselors 


4,000 


n.a. 


School support 


2,000 


n.a. 


Charter School Consortium 


Principals 


20,000 


15,000 


Assistant principals 


15,000 


10,000 


All instructional staff 


1,500 


750 



a DC’s EPIC program — called TEAM — does not include Silver-Gain schools. 

In addition to the school-wide awards made in Gold-Gain and Silver-Gain schools, in 
2008/2009 and later years, EPIC will make “Spotlight Teacher” awards to selected teachers 
in the Gold-Gain schools. Spotlight Teachers will be identified through a combination of 
teacher-level VAM estimates, classroom observations and interviews with building staff. 

Table 1.3 shows how the timing of awards translates into incentives. A given year’s 
incentive payments are based on growth in student achievement during the previous school 
year. So in 2008/2009 teachers will receive an award based on their 2007/2008 performance. 
Performance during the 2007/2008 school year is measured using VAM models. These 
models compare test score performance at the end of that school year with test score 
performance at the end of the previous school year, adjusting for factors beyond the control 
of the school. 



Table 1.3. Teacher Performance and Incentive Payments 



Principals and Teachers Have 
the Incentive to Improve Student 
Performance in Year... 


...to Receive 
an EPIC Award 
in Year 


n.a. 


2007/2008 


2007/2008 


2008/2009 


2008/2009 


2009/2010 


2009/2010 


2010/2011 


2010/2011 


2011/2012 
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Effective Practices 

The second key feature of EPIC is the documentation and dissemination of effective 
practices. Once the Gold-Gain schools have been identified, NLNS will work to identify the 
effective teachers in those schools. In general, the effective teachers will be identified 
through a combination of teacher-level VAM estimates, classroom observations and 
interviews with building staff and will receive “Spotlight” teacher awards. However, in the 
2007/2008 school year, effective practices will be identified at Gold-Gain schools without 
awarding Spotlight Teachers. 

NLNS will send Effective Practice Teams into the classrooms of the Spotlight Teachers 
in the Gold-Gain Award schools to document effective practices employed by these 
teachers. The practices will be identified based on previous research and through classroom 
observations, interviews with the Spotlight teachers, and interviews with other building staff. 

Once the effective practices are identified and documented, NLNS will develop a 
dissemination campaign. Effective practices dissemination will not be targeted exclusively at 
those schools eligible for an award. NLNS plans to disseminate practices not only district 
wide but also nation wide. Current plans call for dissemination through internet-based 
presentations and videos but NLNS is also considering other modes of dissemination. 

Conceptual Framework 

Both the incentive payment and the effective practice components of EPIC can lead to 
increases in student performance (Figure 1.1). First, the availability of a performance-based 
award may alter teacher effort for those who are motivated to receive an award. Teachers 
may work more intensely with students to improve their performance, and/ or they may be 
more willing to try new instmctional practices that affect student performance. Second, 
regardless of the incentives, the effective practices disseminated by NLNS may change 
principal and teacher practices in ways that improve student performance. 

Our evaluation will be focused on two key relationships in this conceptual framework. 
First, we will examine whether there is a relationship between a school’s eligibility for 
incentive awards and student performance. Second, we will examine whether principals and 
teachers alter their practices in response to the effective practice dissemination. Because the 
dissemination of effective practices is not exclusive (leaving us without an appropriate 
comparison group), we will not be able to examine whether any changes in principal or 
teacher practices caused by the EPIC dissemination activities lead to changes in student 
performance. 
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Figure 1.1. Conceptual Framework for the Effects of EPIC on Student Performance 




Research Questions 

The evaluation will focus on both the incentive award component and the effective 
practices component of EPIC. Separate research questions have been developed for each of 
those components. This section presents the research questions. The subsequent sections 
then present our approach to answering these questions, and our approach to collecting the 
data needed to answer these questions. 

Incentive Components 

It is important to note that the set of research questions for this study is limited in part 
by constraints on our ability to estimate the counterfactual. In examining the impact of 
incentives, we would like to know how students would have performed if their teachers were 
not eligible for an incentive award, but the lack of an appropriate comparison group for 
some (but not all) partners means we cannot estimate that counterfactual. As a result, for 
partners where we have no comparison group, our research will simply examine the degree 
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to which principals and teachers are aware of their eligibility, but not whether eligibility 
affects student performance. In those districts where there is a comparison group, we will 
examine how student performance is affected by EPIC eligibility. 

We will use both quantitative and qualitative procedures to answer the following 
questions: 

• How aware are principals and teachers that they are eligible for EPIC’s 
performance based incentives? Do principals and teachers in schools that are 
ineligible for EPIC have the misconception that they are eligible? 

• What are principals’ and teachers’ perceptions of performance-based pay? How 
do these perceptions change over time? 

• In what ways do teachers and school administrators learn of the availability of 
these incentives? 

• Do principals understand the requirements associated with eligibility for EPIC 
incentive payments? 

• What are principals’ expectations regarding whether incentives will affect 
teacher behavior and student outcomes? 

• What impact does eligibility for EPIC incentive awards have on student 
achievement? (This item was only examined for partners with a comparison 
group.) 

• Do the answers to these questions differ by partner or, when appropriate, by 
student or staff characteristics? 

Qualitative procedures will be used to answer the following questions: 

• Are there other concurrent changes that could explain changes in student 
outcomes? 

• What successes and challenges were encountered in the implementation of the 
incentive award components of EPIC? 

• In what ways do school staff believe that the incentive components of EPIC 
could be improved? 

Additional quantitative questions will be examined as part of exploratory analyses. In 
particular, we will examine whether there are differences in impacts by school characteristics, 
and we will test the robustness of the impact estimates by using various subsets of the 
comparison groups. Our statistical power will be low for these types of questions. 
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Effective Practices 



The effective practices identified by NLNS will be disseminated to all schools, 
regardless of whether the school is eligible for incentives or even part of a partner 
organization. As a result, we have no comparison group with which to examine whether 
student performance in schools that receive effective practice information differs from 
student performance in schools that do not receive effective practice information. We can, 
however, examine whether principals and teachers change how they use the practices that 
are disseminated by EPIC relative to how they change practices not disseminated by EPIC. 

We will use both quantitative and qualitative procedures to answer the following 
questions: 

• Before EPIC’s best practices are disseminated, how do principals and teachers 
learn about best practices for teaching in their grade/ subject? 

• Before EPIC’s best practices are disseminated, what are principals’ and teachers’ 
expectations for teacher practices disseminated via the internet? 

• How do EPIC Effective Practice Teams identify best practices? How are they 
disseminated? 

• What are principals’ expectations regarding whether EPIC effective practices 
will lead to changes in principal and teacher behavior? 

• Are principals and teachers aware of EPIC effective practices after they are 
disseminated? How do principals and teachers become aware of these practices? 

• How do principals and teachers report changing their teaching practices in 
response to EPIC effective practice dissemination? 

• Are schools with NLNS-trained principals more likely to adopt the NLNS- 
disseminated EPIC effective practices than schools without NLNS principals? 

• Do the sources that principals and teachers use to identify best practices change 
after EPIC effective practices are disseminated? 

• Are principals with positive expectations for internet-based practices more likely 
to adopt EPIC effective practices? 

• Are principals or teachers with positive attitudes toward performance-pay 
programs more likely to adopt EPIC effective practices? 

• Does the use of the educational practices disseminated by EPIC increase more 
than the use of other types of educational practices known to be effective after 
the EPIC effective practice dissemination occurs? 
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• Do schools eligible for EPIC awards adopt NLNS effective practices more than 
other schools? 

• Do the answers to these questions differ by partner or, when appropriate, by 
student or staff characteristics? 



Qualitative procedures also will be used to answer these questions: 

• Are there other concurrent changes that could affect the practices adopted by 
principals and teachers? 

• What successes and challenges were encountered in the implementation of the 
effective practice component of EPIC? 

• In what ways do principals and teachers believe that the effective practice 
components of EPIC could be improved? 

The answers to these research questions will be used in both formative and evaluative 
ways. Interim results on principals’ awareness of and expectations for EPIC will be used by 
NLNS to refine their information-dissemination approach. Analysis of changes in student 
performance (where possible) and changes in principal and teacher behavior will be used to 
understand the impact of EPIC. 



Research Approach 

This section describes the approach we will use to answer research questions on the 
incentive award and effective practice components of EPIC. While the subsequent section 
provides the details on how we will collect data to conduct this research, it is useful now to 
list the data sources for this study. Four primary sources of data will be used to answer the 
research questions: 

1. Administrative data on student test performance and background 
characteristics 

2. Surveys of principals and assistant principals in EPIC-eligible and EPIC- 
ineligible schools conducted in spring of 2008 and the spring of 2010 

3. Surveys of teachers in EPIC-eligible and EPIC-ineligible schools conducted in 
the spring of 2010 

4. Case studies of EPIC-eligible schools conducted during the 2008/2009 and 
2009/2010 school years 



Table 1.4 indicates which sources of data will be used to answer the various research 
questions in this study. 
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Table 1.4. Research Questions by Data Source 



Research Question 


Student 

Data 

(VAM) 


Principal 

Survey 

(2008 

and 

2010) 


Teacher 

Survey 

(2010) 


Case 
Studies 
(2009 and 
2010) 


EPIC Incentive Awards 










How aware are principals and teachers that they are eligible for EPIC’s performance-based incentives? 
Do principals and teachers in schools that are ineligible for EPIC have the misconception that they are 
eligible? 




S 


S 


S 


What are principals’ and teachers’ perceptions of performance-based pay? How do these perceptions 
change over time? 




S 


S 


S 


In what ways do teachers and school administrators learn of the availability of these incentives? 




S 


V 


S 


Do principals understand the requirements associated with eligibility for EPIC incentive payments? 




S 




S 


What are principals’ expectations regarding whether incentives will affect teacher behavior and student 
outcomes? 




S 




S 


What impact does eligibility for EPIC incentive awards have on student achievement? (Only examined 
in districts with a comparison group) 










Are there other concurrent changes that could explain changes in student outcomes? (Only examined 




S 


s 


S 



in districts with a comparison group) 

What successes and challenges were encountered in the implementation of the incentive award 
component of EPIC? 

In what ways do school staff believe that the incentive components of EPIC could be improved? 

EPIC Effective Practices 

Before EPIC’s best practices are disseminated, how do principals and teachers learn about best 
practices for teaching in their grade/subject? 

Before EPIC’s best practices are disseminated, what are principals’ expectations for teacher practices 
disseminated via the internet? 




Table 1.4 (continued) 



Research Question 


Student 

Data 

(VAM) 


Principal 

Survey 

(2008 

and 

2010) 


Teacher 

Survey 

(2010) 


Case 
Studies 
(2009 and 
2010) 


How do EPIC Effective Practice Teams identify best practices? How are they disseminated? 










What are principals’ expectations regarding whether EPIC effective practices will lead to changes in 
principal and teacher behavior? 




S 






Are principals and teachers aware of EPIC effective practices after they are disseminated? How do 
principals and teachers become aware of these practices? 




S 


S 




How do principals and teachers report changing their teaching practices in response to EPIC effective 
practice dissemination? 




S 


S 




Do the sources that principals and teachers use to identify best practices change after EPIC effective 
practices are disseminated? 




s 


S 




Are schools with NLNS-trained principals more likely to adopt the NLNS-disseminated EPIC effective 
practices? 




s 






Are principals with positive expectations for internet-based practices more likely to adopt EPIC 
effective practices? 




s 






Are there other concurrent changes that could affect the practices adopted by principals and teachers? 




s 






What successes and challenges were encountered in the implementation of the effective practice 
component of EPIC? 










Are principals or teachers with positive attitudes toward performance-pay programs more likely to 
adopt EPIC effective practices? 




s 


S 




Are the educational practices disseminated by EPIC used more than other types of educational 
practices known to be effective after the EPIC effective practice dissemination occurs? 




s 


S 




Do schools eligible for EPIC awards adopt NLNS effective practices more than other schools? 




S 


S 




In what ways do principals and teachers believe that the effective practice components of EPIC could 
be improved? 




s 
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Incentive Awards 

Probably the most difficult-to-answer question about the incentive awards is whether 
the availability of incentive awards leads to changes in student outcomes. To answer this 
question, we will compare changes in student achievement at the “treatment” schools (that 
is, EPIC-eligible schools) with some proxy for what changes would have been observed in 
the absence of incentives. 

One proxy for what would happen in the absence of incentives is what happens to 
outcomes over the same pre-EPIC to post-EPIC period at “comparison” or “control” 
schools. To hold constant such factors as teacher labor markets and administrative policies, 
the comparison schools should be schools that are similar to the treatment schools but 
ineligible for EPIC. In our case, we will look at schools that are within the same school 
district . 4 Ideally, we would identify comparison schools through random assignment. That is, 
treatment (EPIC-eligible) and comparison schools (EPIC-ineligible) would be selected at 
random from within the same school district. However, the partners were not amenable to 
random assignment, partly for logistical reasons and partly because of concerns about trying 
to simultaneously explain merit pay and random assignment to educators. 

Without random assignment, a quasiexperimental design (QED) can be used. Under a 
QED, treatment and comparison schools are selected by a process that is not random. It is 
still important that treatment and comparison schools are similar, and it is crucial that the 
comparison schools do not receive the treatment (that is, do not believe they are eligible for 
EPIC awards). Given that any differences in outcomes could be attributed to the selection 
process instead of the treatment, the results from a QED are generally less compelling than 
similar results from a study using random assignment. 

EPIC was implemented in different ways in the three partners in our study. As a result, 
the opportunity to identify a comparison group differs from district to district: 

• In Memphis, there are both EPIC-eligible and EPIC-ineligible schools. While 
the schools are different in important ways in addition to EPIC eligibility status, 
it is still possible to estimate an impact of incentive awards by looking at how 
outcomes changed for eligible schools compared with ineligible schools when 
EPIC was implemented. 

• In DC, virtually all schools are eligible for EPIC incentive awards. As a result, 
there is no comparison group of schools with which to measure an impact of 
incentive awards. 



4 An explicit assumption in using comparison schools from the same district is that there is no 
“contamination” of the treatment in the comparison schools. However, the potential for contamination exists. 
One type of contamination would occur if comparison schools believed they were eligible for an award. 
Another type of contamination would occur if high-performing teachers from ineligible schools transferred to 
eligible schools (affecting the teacher composition of both the eligible and ineligible schools). 
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• In the Charter School Consortium, there are no schools that are ineligible for 
EPIC. Nevertheless, there are a handful of cities with multiple EPIC-eligible 
charter schools and also large numbers of EPIC-ineligible charter schools. In 
those cities, it may be possible to obtain data from a comparison group of 
EPIC-ineligible charter schools for use in our study. 

Where viable comparison groups exist, we will employ a difference-in-differences 
approach to estimate the impact of EPIC eligibility on student performance and other 
outcomes. The difference-in-differences approach examines whether changes observed at 
treatment schools before and after EPIC are different from similarly measured changes at 
comparison schools. For example, if student performance increases more at treatment 
schools relative to comparison schools, this will be taken as evidence that EPIC incentive 
awards improved student performance. 

For the Charter School Consortium and for Memphis, a difference-in-differences 
approach can be employed. The specifications of the separate models used to estimate the 
difference-in-differences are discussed in the subsequent chapters for each of those partners. 
For DC, there is no valid comparison group of schools. As a result, we cannot estimate the 
impact of EPIC incentive awards there. 

Other, nonimpact questions about the incentive awards can be answered for all 
partners. For example, we can examine principal and teacher responses to surveys and case 
study interviews to gauge their level of awareness of EPIC to determine whether the 
treatment itself even occurred. We can also explore principal and teacher perceptions of 
performance based incentives and whether these perceptions change over time. Also some 
measures of principal and teacher behavior and perceptions can be compared across 
partners. It is possible that staff awareness of EPIC and/or their opinions of performance- 
based awards could differ from partner to partner, perhaps in part because of variations in 
how the EPIC incentive components were implemented with the different partners. 

Effective Practices 

Our analysis of the effective practices component of EPIC is focused on two issues. 
First, we will conduct a process analysis of the identification of and dissemination of 
effective practices. Using a case study approach, we will shadow NLNS staff as they identify 
effective practices, and interview staff at the district and school level. Our case study 
approach will document the steps NLNS takes to identify and disseminate effective 
practices, and will discuss key challenges (and the implications of those challenges) 
experienced along the way. 



I: Introduction 




14 



The second question about the EPIC effective practices is whether principals and 
teachers adopt those that are disseminated by EPIC. 5 Effective practices are disseminated to 
all schools, regardless of their eligibility for EPIC incentive awards. As a result, there is no 
control group with which to test whether staff in schools receiving EPIC effective practices 
dissemination change practices in ways different from staff in schools that did not receive it. 

Instead, we can examine a broad array of baseline practices of principals and teachers in 
all schools in a district (regardless of incentive award eligibility), and then see if those staffs 
report larger increases in practices that are disseminated by EPIC than in practices not 
disseminated by EPIC. This will allow us to examine whether effective practice 
dissemination increases the likelihood that schools change some practices versus others. 
Additionally, for partners that have both eligible and ineligible schools, we can examine 
whether schools that are eligible for an EPIC award are more likely to adopt EPIC effective 
practices. 

A key assumption in our analysis of how principals and teachers respond to effective 
practice dissemination is that the level of dissemination is uniform across schools in our 
analysis sample. Thus, schools have equal access to the information that is disseminated 
regardless of whether they are eligible for EPIC, win an EPIC award, have a principal trained 
by NLNS, etc. 

The fact that the EPIC effective practices have not yet been defined facilitates this 
design because it ensures that principals and teachers are unaware of which effective 
practices EPIC will disseminate. However, it also complicates the design because it requires 
us to measure a wide set of practices at baseline and hope that some, but not all, of those 
captured in the surveys end up being identified by EPIC as effective. If all or none of the 
practices are disseminated by EPIC, this design will not be able to measure changes in the 
use of effective practices (and we will be able to gauge changes only through post- 
dissemination self-reporting of principals and teachers). 

To develop a set of baseline practices, we worked with EPIC staff to identify the types 
of practices in which they have the greatest interest. These include such practices as the use 
of data to track school and class outcomes in a formative way, the use of structured staff 
meetings to share practices, etc. We will measure these baseline practices both in terms of 
the relative importance teachers and principals place on them (not important to very 
important) and the frequency with which they are employed (never to always). This will 
allow us to identify situations where staff employ practices that they believe are unimportant 
(and therefore may not implement them effectively) as well as situations where staff are not 
using practices that they believe are important. We can then track changes along both 
dimensions. 



5 The effective practices and the incentive awards are interrelated. While we treat the components as 
separate for the purposes of describing the evaluation design, in practice there may be a relationship between 
whether a school is eligible for an award and whether they adopt the effective practices disseminated by NLNS. 
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Due to funding limitations, our baseline survey is limited to principals and assistant 
principals. We can examine changes in principal practices by comparing principal responses 
in baseline to principal responses in the follow-up survey. We can also explicitly ask 
principals in the follow-up survey about changes they may have made in response to EPIC 
effective practice dissemination. Without a baseline survey for teachers, we can gauge 
changes in teacher practices only by (1) examining changes in the practices that principals 
report their teachers are using and (2) by asking teachers whether they have changed 
practices in response to EPIC effective practice dissemination. 

Other, nonimpact questions about the effective practices component will also be 
examined. Specifically, we can examine principals’ expectations for the effective-practices 
component of EPIC before the practices are disseminated, and examine how aware 
principals and teachers are of the practices after they have been disseminated. We can also 
use the information collected through case studies to further explore how principals and 
teachers respond to the EPIC effective practices. The case studies will allow us to talk with 
staff in detail about why they may or may not find the EPIC effective practices relevant or 
useful. We can use the case studies to identify and illuminate dissemination strategies. We 
also can use the case studies to identify obstacles to effective dissemination. 

Data Collection 

Three key types of data will be collected for this study: (1) data on student 
characteristics and academic performance, (2) survey data, and (3) site visit data. 

Student Data 

Student data will be collected for VAM estimation. For each partner, data will be 
collected for 2005/2006 through 2010/2011 so NLNS can identify Gold-Gain and Silver- 
Gain awards for each partner. For the evaluation of EPIC, we will use data from 2005/2006 
through 2009/2010. 

The information available on student records differs from partner to partner (Table 1.5). 
These data will be used to describe the characteristics of schools in each partner (where 
relevant, comparing treatment and comparison schools at baseline). They also will be used to 
examine the impact of incentive awards on student performance. 

We will conduct a thorough analysis of these data for each of the partners, as will be 
discussed in subsequent chapters. We will prepare a series of detailed tables comparing 
school characteristics across meaningful groupings to identify any important baseline 
differences that could influence difference-in-differences comparisons, survey response 
patterns, or other components of this evaluation. The specific characteristics that are 
examined are a function of the student information available from each partner. The 
groupings across which comparisons will be made will depend in part on the characteristics 
of EPIC in each partner. 
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Survey Data 

Survey data will be used to answer research questions for both the incentive awards and 
the effective-practices components of EPIC as shown in Table 1.4. Two rounds of surveys 
will be conducted. In the first round, to be conducted in the spring of 2008, we will survey 
principals and assistant principals (APs) in each of the three partners. The survey will be 
administered to a sample of principals and APs in schools that are eligible for the incentive 
awards, and, in Memphis, also to a sample of principals and APs in schools that are ineligible 
for the incentive awards. 6 A total of 80 schools will be selected for the survey in Memphis 
and DC, as well as all 97 schools in the charter consortium. All principals and APs in those 
schools will be included in the sample. 



Table 1.5. Student Information Available by Partner 





Memphis 


DC 


Charter School 
Consortium 


Grade 


■/ 


■/ 




Academic performance 


■/ 


■/ 


■/ 


Days enrolled 








Days present 


■/ 






Race/ethnicity 


■/ 


■/ 


■/ 


Gender 


■/ 


■/ 


S 


Free/reduced price lunch eligibility 


■/ 




S 


Limited English proficiency status 


■/ 


■/ 


S 


English as a second language status 




■/ 




Special education status 


■/ 




S 


Suspended days 


■/ 






Student retained 


■/ 


■/ 




Expulsion status 


■/ 






Excused and unexcused absences 




■/ 





The second round of the survey, to be conducted in the spring of 2010, will be 
administered to the same schools in each of the partners. Surveys will be administered to the 
principals and APs, and a separate survey will be administered to a sample of 5 teachers in 
each of the 80 schools. 



6 The principal/ AP survey in 2008 is not being given to the comparison charter schools because those 
schools have not yet been identified. 
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Because there is no survey of teachers conducted in the spring of 2008, information on 
whether teacher practices change in response to EPIC will be obtained from two sources. 
First, in the 2010 teacher survey, we will ask teachers to self-report changes in practices. Self- 
reported changes are helpful but are subject to bias if teachers are more likely to report 
changes in certain practices simply because they are being disseminated by EPIC. Second, we 
will ask principals to generalize about teacher practices in 2008 and again in 2010, and we 
will examine whether principals’ reports of teacher practices change between 2008 and 2010. 
This measure of change is constrained by whether principals can accurately report on teacher 
practices and whether the range of teacher practices in a given school is narrow enough to 
be easily generalized. For each sampled school, we will test both of these constraints by 
comparing principals’ reported teacher practices in 2010 with teacher reported practices in 
2010. If the principal and teacher reports are consistent, we can have more confidence in 
changes measured from principal reports in 2008 and 2010. 

Instrument. The instrument for the spring 2008 Principal/ AP survey was designed to 
capture the following information: 

• Awareness of EPIC, including eligibility requirements and award criteria 

• Perceptions of performance-based pay 

• Current practices of principals, APs and teachers 

• Current sources of best practices for principals, APs and teachers 

• School environment 

• Other changes concurrent with EPIC 

Appendix A includes the table of constructs captured in the survey and Appendix B 
includes the survey instrument for the spring 2008 Principal/ AP survey. The instmment is 
intended to take 25 to 30 minutes to complete. The survey will be pretested on a sample of 
principals from schools that are not part of the partners. 

The primary mode for this survey will be the internet. A paper instrument will be mailed 
to principals and APs, but the cover letter accompanying the paper instmment will 
encourage completion over the internet. For principals and APs who do not complete the 
internet or paper version, we will attempt to conduct the survey by phone. The telephone 
follow-up has been proven effective in increasing response rates. Additionally, we will offer 
all sampled principals and teachers incentives (in the form of $25 checks) to participate in 
the survey. We anticipate achieving an 85 percent response rate on each survey, with three 
quarters of the questionnaires completed by mail or internet and one quarter by telephone. 

Similar instmments will be developed for the 2010 survey. The instruments will capture 
much of the same information on current practices, as well as information on the reason for 
changes in practices and on perceptions of EPIC effective practice dissemination activities. 
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Selecting Schools and Sample Members. For each partner, we will construct a 
sample frame of schools for the survey. In Memphis and DC, 80 schools will be randomly 
sampled. The sample will be stratified by eligibility for EPIC incentives (where relevant), by 
the level of the school (elementary, middle or high), and by whether the principal at the 
school was trained by NLNS. Proportionate sampling will be conducted within each stratum. 
For the Charter School Consortium, all 97 schools in the consortium in the 2007/2008 
school year will be included in the survey. 

Case Study Data 

Case study data will be used to enhance our knowledge of how the EPIC initiative is 
implemented. Case studies will be central in examining the process of identifying and 
disseminating effective practices. While case study data will not be used directly in the 
impact-estimation models, the results of the case studies will be critical in helping us 
understand where and how EPIC may have an effect on teacher practice and student 
performance, and will enable us to check the face validity of our results. Case study site visits 
will be used to develop a deeper understanding of educators’ awareness, understanding, and 
perceptions of EPIC than would be possible using the survey data alone. The case studies 
will also be used to document the methods that Effective Practice Teams use to assess the 
practices of effective teachers and to disseminate those effective practices. 

Two rounds of case studies will be conducted at four schools for each partner. The first 
round will be in the spring of 2009. At this point, EPIC incentive awards will have been 
made twice (with the second award occurring the previous fall). Additionally, the NLNS 
Effective Practice Teams will have conducted two rounds of their own case studies to 
identify effective practices, and will have disseminated the practices identified through their 
first round. The second round of MPR case study site visits for this evaluation will occur in 
the spring of 2010, after three rounds of awards, three rounds of effective practice 
identification and two rounds of effective practice dissemination. The 2010 case study site 
visits will occur before the 2010 principal and teacher surveys. 

Selecting Schools. Since each partner will have more than four schools participating in 
the EPIC intervention (i.e., eligible for awards), MPR will select a subset of those schools for 
the case study component. Schools that are ineligible for incentive awards will not be 
included in our site visits. 

For the first round of case studies, we will work with NLNS on an appropriate sampling 
plan based on the needs of our evaluation. The decision on which characteristics to use in 
selecting schools will depend in part on NLNS experience with identifying effective practices 
in the 2007/2008 school year and on how NLNS identifies award-winning schools in the 
2008/2009 school year. One option is to sample schools based on grade levels served to 
help determine how impacts of EPIC might vary in this dimension. We could also sample to 
ensure we obtain at least some schools that had received an EPIC gold award and thus 
participated in the collection of effective practice information. A site visit with a school with 
a New Leaders principal might help inform us about how EPIC can work when New 
Leaders has a strong connection to the school. Finally, we may be able to use self-assessment 
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data that school administrators provide to NLNS as part of EPIC to help identify other 
useful selection criteria. 

The second round of case studies may or may not be conducted with the same schools 
visited during the first round. This will depend in part on the results of the first survey and 
the first round of site visits, and on any developments in the implementation of EPIC over 
time. If additional research questions arise for which it would be useful to know how the 
same schools are responding to new issues, we will visit the same four schools. If, however, 
that is not the case, we will select four new schools to visit during the second round of case 
study site visits. 

Conducting Site Visits. Two MPR team members will jointly conduct the visits to 
each school district. The first activity in each district will be a meeting with the district 
superintendent or his or her designee (for charter schools, we may conduct a meeting with 
the relevant charter management organizations, as appropriate). During this meeting, the site 
visitors will introduce themselves and the purpose of the visit. They also will meet with the 
district’s NLNS liaison during that initial visit. 

The school-level visits will have similar agendas. Site visitors will interview the principal, 
assistant principal(s), and teachers. Most of these interviews will be one-on-one, except two 
discussions will be held with small groups of teachers. 

We will ask each school principal to designate a staff person to help us schedule the site 
visit. We will randomly select five teachers for the group-level discussions and three teachers 
for individual interviews and will stratify our sample to ensure that at least some Spotlight 
Teachers are included in our sample in Gold-Gain schools. We will sample so that across all 
of the schools in each partner the identified teachers vary by grade level, subject taught, and 
years of experience. We will work with the designee on how to best accommodate teachers’ 
schedules so that we can conduct the group discussions and one-on-one interviews. If 
scheduling difficulties arise we will replace unavailable teachers with randomly selected 
available teachers and ask principals if the new teachers selected differ from those who were 
unavailable. 

In addition to interviews with district and school staff, MPR staff will shadow the 
Effective Practices team in some schools. In this shadowing process, MPR staff will observe 
(but not participate in) meetings with staff of Gold-Gain schools to identify and articulate 
effective practices in those schools. We will use this information to better understand exactly 
how effective practices are identified. 

For each round of site visits, we will develop a semi-stmctured protocol to guide 
interviews with district and school staff. Topics covered in the protocol may include: 

• Teacher Best Practices. Flow do principals and teachers currently identify 
effective teaching practices to use in the classrooms? What are the resources 
they most commonly use? Why? What kinds of resources would they like to 
have available? 
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• Rollout of EPIC. How were the schools and teachers informed about EPIC? 
When did they hear that NLNS and the district would be implementing EPIC? 
When and how did schools and teachers learn about the award announcements? 

• Awareness of EPIC. What do respondents know about EPIC, including the 
incentives for student achievement and the identification and dissemination of 
effective practices? Do they know the basis on which the awards are made? Are 
staff more aware of some aspects of EPIC than others? 

• Identification of Effective Practices. For Gold-Gain schools, what steps did 
NLNS take to identify effective practices? How did school staff, district officials 
and/ or NLNS determine that the practices are, in fact, effective? 

• Dissemination of Effective Practices. What steps did NLNS take to 
disseminate effective practices? Do school staff find the information 
disseminated helpful? Do staff find the dissemination venues accessible? Do 
staff at Gold-Gain schools feel that the information disseminated accurately 
reflects the practices they believe are effective? 

• Perceptions of EPIC. What are respondents’ reactions to EPIC? What do they 
think are the benefits? What do they think are the drawbacks? How do they 
think the EPIC will affect their behavior in the classroom? How do they think it 
will change their colleagues’ behavior? In what ways could EPIC be improved? 

• Quality of School Climate. What factors influence the school climate, and 
how? To what extent do administrators and teachers enjoy working at the 
school? What is the average tenure of an administrator at the school? A teacher? 

Can this information be better obtained from administrative data? 

The protocol will be used as the basis for a template for writing up the case studies. As 
soon as possible after completing each site visit, the site visit team will prepare a report, 
conforming to this template, and summarize the responses of the various respondents to 
each of the protocol questions. 

Analysis of Site Visit Data 

Upon completing the site visits and site visit reports, we will conduct an analysis of 
information collected during our visits. This analysis will be used to identify common themes 
emerging in response to the various questions explored on the site visits. The analysis will be 
conducted in three key steps: 

Step 1: Summarize Information by Interview and School. The first step in analyzing 
site visit data will be to summarize information from each interview or groups of interviews. 
For each partner, we will summarize all information collected through the various interviews 
at each school. Table 1.6 shows a sample grid for a theme table examining school 
information. For each partner, one theme table will be created to summarize information 
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collected during district-level interviews, one to summarize the results of interviews with 
principals and APs at each school visited (Table 1.6) and one to summarize the results of 
interviews with teachers at each school visited. The tables can then be reviewed to identify 
common threads. 

Table 1.6. Grid for Site Visit Theme Table: Partner A 

Principal and Principal and Principal and Principal and AP, 
Theme AP, School A AP, School B AP, School C School D 

Teacher best 
practices 

Rollout of EPIC 

Awareness of 
EPIC 

Perceptions of 
EPIC 

Quality of school 
climate 



Step 2: Summarize Information Across Data Collection Activities. We also will 
create theme tables to allow us to compare information collected via site visits with 
information collected through the surveys. For each partner, we will distill all site visit 
information collected from district staff, principals, and teachers. Where relevant, we will 
summarize the pertinent information collected through our principal and teacher survey. 
These will be presented in a second -level theme table constructed for each partner (Table 
1.7). This will allow us to identify areas where site visit and survey data are consistent and 
where they are inconsistent. 

Step 3: Describe the Teacher Practices and Perceptions of EPIC. After the theme 
tables have been completed, we will develop a comprehensive, detailed description of 
teacher practices and teacher perceptions of EPIC. These descriptions, which will be 
partner-specific, will summarize the various themes emerging from our analysis of the theme 
tables. We will characterize the four schools visited in our site visits and compare those 
themes with results from our survey. 
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Table 1.7. 


Grid for All Data Collection Activities: Partner A 




Theme 


Site Visit Interviews 


Survey 



Teacher best practices 
Rollout of EPIC 
Awareness of EPIC 
Perceptions of EPIC 
Quality of school climate 



Schedule 

Table 1.8 presents the general schedule for the evaluation of EPIC. The initial incentive 
awards were made between December 2007 and March 2008. In the spring of 2008, we will 
conduct our first principal/ AP survey, and NLNS will begin identifying effective practices 
through visits to Gold-Gain schools; Silver-Gain schools will be asked to provide written 
documentation of their practices but will not be visited. In fall 2008, a second set of 
incentive awards will be made. In spring 2009, NLNS will begin identifying effective 
practices through visits to the new award-winning schools in both the Gold-Gain and Silver- 
Gain tiers, and MPR will conduct the first round of case study site visits. A similar pattern 
will occur in the 2009/2010 school year, beginning with the third round of awards in fall 
2009. In spring 2010, NLNS will conduct the third round of effective practice visits and 
MPR will conduct the second round of case study site visits and administer the principal/AP 
and teacher surveys. 

Outline of the Remainder of the Report 

This chapter has provided an overview of the design for evaluating the incentive award 
and effective practice components of EPIC. The subsequent chapters discuss specific design 
details for Memphis, DC, and the Charter School Consortium, respectively. In each chapter, 
we describe partner-specific details on the EPIC initiative, explain how impact estimates 
(where possible) will be computed, and discuss special considerations in sampling and 
analysis. 
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Table 1.8. Evaluation Schedule 



Timeframe 


Activity 


Winter/spring 2008 


First EPIC incentive awards 


Spring 2008 


Baseline principal survey 

Effective practice identification visits by EPIC team 


Fall 2008 


Second EPIC incentive awards 


Spring 2009 


First MPR case study visits 

Effective practice identification visits by EPIC team 


Summer 2009 


Estimates of impact of incentive payments on student 
performance in 2008/2009 


Fall 2009 


Third EPIC incentive awards 


Spring 2010 


Follow-up principal survey 

Teacher survey 

Second MPR case study visits 


Summer/fall 2010 


Report on EPIC effective practices 
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Chapter 1 1 



Evaluation of EPIC in Memphis 



I n this chapter, we describe the criteria and results for the first round of EPIC awards for 
Memphis City Schools (MCS), the eligibility requirements for those awards, and how we 
will compare background characteristics of eligible and ineligible schools. Next, we 
explain how MPR will evaluate the incentive award components of EPIC for MCS, with a 
particular focus on the quantitative analyses. Finally, we discuss the sampling criteria for 
schools chosen to participate in the principal and teacher surveys. 



MCS 2007 Awards 

In early 2007, New Leaders for New Schools (NLNS) informed MCS of their eligibility 
for school performance awards, the first of which were given out by New Leaders in 
December 2007. The awards were based on value-added performance for the 2006/2007 
school year as calculated by MPR. NLNS presented 5 schools with Gold-Gain awards and 
12 schools with Silver-Gain awards. The recipients of the Gold-Gain awards, which included 
2 elementary schools, 2 middle schools, and 1 high school, had the greatest value added 
scores. The 12 Silver-Gain schools (10 elementary, 1 middle, and 1 high school) ranked just 
below the gold schools in terms of value-added scores.' This is out of a total of 136 
Memphis schools that were eligible for EPIC awards. 

In the spring of 2008, NLNS will start to visit EPIC award-winning schools in Memphis 
to identify promising practices and gauge educators’ reactions to EPIC. The next school 
awards for Memphis will be announced in 2008/2009 on the basis of VAM performance as 
measured in spring 2008 tests. Spotlight Teacher awards will be given out in Memphis 
starting in 2009/2010. 

Eligibility for an EPIC award in Memphis depends on several factors. To be eligible, a 
school (1) must have at least 50 percent of its student body qualifying for free or reduced- 
price lunch (F/RPL), (2) cannot be a charter school, and (3) cannot be participating in 



7 An additional five schools were nominated for an award; however, since less than 80 percent of the 
teachers in these schools voted to accept the award, as required in Memphis, the schools did not receive the 
awards. New Leaders plans to investigate why teachers chose to reject the awards in those schools. 
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another program that offers performance pay. Memphis had two such incentive programs: 
the Fresh Start program and the Striving School program. 

The Fresh Start incentive program, which began in June 2004, rewards staff in 
participating schools for meeting certain performance goals. These goals are based on 
outcomes such as grades, test scores, completion of Individual Education Plans for special 
education students, disciplinary outcomes, student attendance, and staff attendance. These 
measures all differ from the VAM models in that there is no attempt to adjust for factors 
beyond the control of the school, such as prior performance or other background 
characteristics. The average teacher awards for the Fresh Start program range from $500 per 
teacher (for schools that reach at least 50 percent of the goals but less than 60 percent) to 
$3,000 per teacher (for schools that reach at least 90 percent of the goals). 8 In its inaugural 
year (2004/2005), five schools participated in the program. The following year, three 
additional schools joined the program. 

The Striving School program was more recently launched. In the summer of 2007, it 
was announced that a group of MCS schools on the state’s probationary list of “striving 
schools” would receive an additional $9.7 million to support an intensive effort to improve 
students’ test scores and overall performance. These funds would allow an anticipated 15 
schools (4 of which are included among the Fresh Start schools), to institute numerous 
changes including lengthening the school day, hiring additional staff, and providing 
performance-based incentives for principals, teachers and staff. 

Our comparison group will be limited to schools for which MPR has value-added data. 
Within this set of schools, some did not meet the F/RPL criterion, some were either Fresh 
Start or “Striving Schools,” some were charter schools, some were alternative schools, and 
some were ineligible for unknown reasons. This results in a comparison group of 46 MCS 
schools and a treatment group of 136 MCS schools. Schools for which MPR does not have 
value-added data primarily include charter, alternative and vocational schools. 



Evaluation 

This section explains the methodologies that MPR will employ to evaluate the incentive 
and dissemination of promising-practices components of NLNS’ EPIC program in 
Memphis. We begin by describing the descriptive statistics we will use to examine the 
characteristics of the schools. We then explain the model we will use to estimate the impact 
of incentives and discuss several variations of it, explain issues related to the power analysis, 
and present the sampling design for the principal and teacher surveys. 



8 The calculation to determine the percent of goals met is fairly complex, based on nine weighted criteria 
that depend on meeting a goal or, if the goal is not met, examining the gain towards meeting the goal. 
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Descriptive Analysis 

To paint a baseline picture of each of the partners involved with EPIC, a series of 
school, student, and teacher characteristics will be compared across different groupings of 
Memphis schools. As discussed in Chapter I, each partner provided us with student 
demographics and other student-level characteristics. We also will obtain school and teacher 
characteristics from the NCES Common Core of Data and other publicly available sources 
where possible. 

The analysis of baseline characteristics will be used, in part, to identify pre-existing 
differences among schools that could account for trends in student achievement, survey 
response patterns, etc. The primary unit of analysis will be the school. Characteristics of 
students and of teachers will first be calculated at a school level and then averaged across 
schools. The descriptive analysis for Memphis will compare schools grouped in a variety of 
ways. Initially, comparisons will be made across such groupings as: 

• Eligibility status (eligible vs. ineligible) 

• Reason for ineligibility (high-income school, Fresh Start school, Striving School 
or charter school) 

• Award status (Gold-Gain vs. Silver-Gain vs. no award) 

• School level (elementary or secondary) 

• Value-added quartile (top quartiles, middle two quartiles, bottom quartile) 

To explore the data further, we also may compare differences by grouping schools 
based on student demographics, school performance or teacher characteristics. Additionally, 
we will conduct baseline comparisons of schools based on responses to the principal survey. 
For instance, we will compare schools according to whether principals are aware of EPIC, 
and whether principals have an understanding of EPIC eligibility requirements. 

Table II. 1 presents a table shell for comparing schools across these different groupings 
using the student and school information in Memphis. 9 This table shell, which is similar to 
those presented in Chapters III and IV, reflects the specific student characteristics available 
in Memphis. We will use t tests to test for significant differences between the groups. 



9 Note that we do not intend to present all of the various tables suggested by the table shell in III.l; 
rather, we will examine these tables to identify potentially meaningful differences and present those differences 
in our reports. 
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Table 11.1. Table Shell for Examining MCS Schools 

Group A Group B Groups C, D, etc. 

School Characteristics 

School type 

Percent elementary 

Percent secondary 

Percent elementary & secondary 

Percent NCLB AYP pass overall 

NCLB AYP status (e.g., “in need of 
improvement”) 

Baseline value-added quartile 
Top quartile of schools 
Middle two quartiles 
Bottom quartile of schools 

Student Characteristics (Averages) 

Enrollment 

Number of days enrolled 
Number of days attended 

Ethnicity 

Percent black 
Percent Hispanic 

Percent male 
Percent free lunch 
Percent limited English 
Percent special education 
Percent retained 
Number of days suspended 
Number of expulsions 

Teacher Characteristics (Averages) 

Staff size 

Student/teacher ratio 



Estimating Impacts of Incentive Awards 

To estimate the impact of incentive awards on student outcomes in Memphis, MPR will 
employ a difference-in-differences approach. We will estimate the difference between the 
change in school performance for the eligible schools and the change in school performance 
for the ineligible schools when EPIC is introduced. If the incentives matter, we would 
expect this change to be larger for the eligible schools than for the ineligible schools. 
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The model, which will be estimated at the student level, can be represented by: 

Score , = a + * Score, , + f3 2 *T + ft , * P t + ff * T * P t + fh * Controls + s 

where Score t is a measure of student achievement, T is a dummy variable for whether the 
student is in a school that is eligible for an award (=1 if eligible, 0 otherwise), P, equals 1 for 
time periods when awards can have an impact on performance and 0 otherwise, the 
subscript t indicates the school year. Controls include student demographic characteristics, 
and s is the error term. For students in the treatment schools, the association of the score 
with being in the treatment group before the incentives have had an impact (P t =0) is p 2 . ln 
After the incentives have had time to have an impact (P t =l), the association will equal p 2 + 
P 4 . We assume that the main reason these two associations differ is the impact of the 
incentives. Therefore, the estimated impact of the incentives on student growth is the 
difference between these associations, or p 4 . 

Our research design may constrain which schools can be included in our comparison 
group. In particular, we do not want to include schools that had incentive programs starting 
at the same time as EPIC since those schools would also, presumably, experience changes in 
performance caused by the introduction of incentive pay. Since the Fresh Start program 
started before the EPIC program we would not expect performance for those schools to 
change at the time EPIC was introduced. Thus, the Fresh Start schools are candidates for 
inclusion in our comparison group. On the other hand, the Striving Schools program may be 
starting at the same time as EPIC and is also offering performance-based incentives. Thus, 
we may need to exclude the Striving Schools from our comparison group. 

The problems with constructing our comparison group highlight the major weakness of 
the difference-in-differences model. If other factors were changing at the same time as EPIC 
in ways that differ for the treatment and control groups, then the difference-in-differences 
model will not be valid. We have included questions about such changes in our survey and 
will analyze responses to those questions to look for patterns that might suggest such 
differential changes. 

Assuming that we do end up with 46 comparison schools and 136 treatment schools, 
this main model has a minimum detectable difference of 0.085 student-level standard 
deviations. 1 That is, we will be able to identify an effect of the EPIC incentives only if the 
incentives lead eligible schools to increase student achievement by 0.085 standard deviations 
or more. 



10 If treatment and comparison schools were identical before EPIC was enacted in terms of their 
performance (after controlling for the background factors in our model), we would expect f} 2 to equal zero. To 
the extent that this is not true, /? 2 will differ from 0 but our estimates of should still be valid. 

11 This is based on a two-tailed test with 80 percent power and a 5 percent significance level. 
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The power of our statistical tests depends largely on the number of schools included in 
the treatment and comparison groups. This means that when we take subsets of the student 
population that do not affect the number of schools our power will remain fairly stable. 
However, when we look at subsets of the sample based on school characteristics, our 
statistical power will be reduced. We will investigate a number of breakdowns of our results 
by student and teacher characteristics and also do some exploratory analyses based on school 
characteristics. 

We will estimate models based on various subsets of the student population, such as 
free-lunch eligibility and baseline test scores, to establish whether estimated impacts vary for 
these subgroups. We will also conduct exploratory analyses to see how the findings differ by 
type of school — elementary, middle and high schools. Research suggests that, on average, 
students’ year-to-year test score growth is greatest at lower grades and the growth rate 
declines as students progress through elementary and secondary school. 12 Can incentive 
awards have a bigger impact on students’ growth in test scores at the elementary level 
because there is more potential for elementary students to experience bigger jumps in test 
scores? Does the fact that elementary students already achieve larger gains mean that 
incentives have less of an impact the elementary levels? Examining the impacts by school 
type can help us answer these types of questions. 

To examine the robustness of our estimates, we will test to see if our estimates vary 
according to which subset of our comparison group schools we use. For instance, we will 
estimate some models excluding all of the Striving Schools in case they experienced 
important changes during the period EPIC is enacted. We will estimate other models that 
limit our comparison group to comparison schools with 50 percent or more of the student 
body eligible for free or reduced-price lunch based on the argument that these are the 
schools most similar to those eligible for EPIC. Finally, we will estimate models excluding all 
charter schools. Each of these subsets of comparison schools differs from the eligible 
schools in unique ways. If our results differ depending on the comparison group chosen we 
will do further investigations to develop an understanding of why this might be the case. 

Another robustness check will be to estimate the model with fixed effects for each 
student’s “spell” of time in a given school. A spell covers the amount of time a student 
spends at one school. For many students this will cover periods both before and after EPIC 
is introduced. Thus, we can estimate the impact of EPIC even after controlling for these 
spell fixed effects. These spell fixed effects would control for fixed effects associated with 
students as well as fixed effects associated with schools. Comparing the results of the basic 
model with results from a model with spell fixed effects will allow us to examine whether 
student migration between schools over the pre-treatment to treatment periods may be 
biasing results in the basic model. 



12 Hill, Carolyn }., Howard S. Bloom, Alison Rebeck Black, and Mark W. Lipsey. “Empirical Benchmarks 
for Interpreting Effect Sizes in Research.” MDRC Working Papers on Research Methodology, New York, NY: 
MDRC, 2007. 
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Additionally, we can examine how the estimate of the impact of EPIC incentives 
changes over time. This could matter if it takes time for teachers to become fully aware of 
the EPIC incentives and/or alter their teaching strategies. To determine this, we will test 
whether a longer exposure to the treatment changes the effect of eligibility on student 
performance. 

Awards provide an incentive to administrators and teachers only if they are aware of 
them. Therefore, we will also estimate the impact of being in the treatment group on 
awareness of EPIC incentives. For this analysis we will rely on survey data. We will not have 
pre-EPIC survey data, so we will be using a cross-sectional regression model, rather than a 
difference-in-differences model. Also, our sample sizes will be substantially smaller than for 
the test score analyses described above. In particular we expect to have survey data for 
around 60 treatment schools and 20 control schools. This will enable us to estimate impacts 
as small as 37 percentage points in awareness of EPIC incentives. Thus, for example, if 5 
percent of people within the comparison group mistakenly believe they are eligible and only 
42 percent of the people in the treatment group believe they are eligible, we should still be 
able to detect this difference with some confidence. 1 ’ We will estimate logit models to 
account for the binary nature of awareness of EPIC. 

As described in Chapter I, we will also use case studies to develop a deeper 
understanding of how perceptions and awareness of EPIC incentive components might 
impact behavior. During the Memphis site visits, we will pay particular attention to the fact 
that schools were able to vote on whether to participate in EPIC. This voting was not part 
of the EPIC program for DC or the Charter School Consortium. 

Survey Sample 

In selecting schools for the survey, we wish to sample those that are eligible as well as 
those that are ineligible for the incentive awards. Including ineligible schools will allow us to 
examine whether ineligible schools mistakenly believe they are eligible for the awards and to 
examine whether ineligible schools are as likely to adopt EPIC effective practices as eligible 
schools. As a result, the sample frame for Memphis will include all MCS schools for which 
MPR has sufficient data to estimate value-added models. 

The sample in Memphis will be stratified. One stratum will be school eligibility status 
for the incentive awards. Other strata will include the level of the school (elementary, middle, 
or high) and whether the school principal was trained by NLNS (among those schools in our 
Memphis sample frame, 35 have a principal or assistant principal who has been trained by 



13 This is based on a two-tailed test with 80 percent power and 5 percent statistical significance. It is also 
based on a binary outcome with a mean of 0.5. If the mean is larger or smaller, our estimates will be more 
precise. 
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NLNS). 14 Since most of the sample frame consists of eligible schools (136 out of 182), most 
of our survey sample will also consist of eligible schools. This will reduce our statistical 
power somewhat for estimating impacts on awareness compared to a model with half of the 
sample eligible and half not eligible. However, as noted, this should still give us sufficient 
statistical power to estimate impacts on staff awareness of EPIC. We are also interested in 
obtaining reasonably precise estimates of staff perceptions of EPIC in the eligible schools. 
For these types of questions, our proposed method will give us more precision than we 
would get with a 50/50 split between the treatment and comparison groups. 



14 Because there are many different grade configurations in MCS, for our purposes, elementary school 
refers to schools with grades six or below only; high schools include schools with grades nine or above; and 
middle schools are all remaining schools, including K— 8 schools. 
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Chapter III 



Evaluating EPIC in the 
District of Columbia 



I n this chapter, we present background on the EPIC program in Washington, DC; 
explain how we will examine the incentive award components of EPIC in DC; and 
discuss sampling criteria for schools chosen to participate in the principal and teacher 
surveys. 

DCPS 2007 Awards 

In 2007, all traditional public schools in the District of Columbia Public Schools 
(DCPS) system were eligible for EPIC school performance awards. 15 In DC, these awards 
are referred to as Together Everyone Achieves More (TEAM) awards. In December 2007, 
Mayor Adrian Fenty and Schools Chancellor Michelle Rhee presented TEAM awards to 
three elementary schools. These schools were chosen on the basis of school-level increases 
in the percentages of the student bodies reaching proficiency between the 2005/2006 and 
2006/2007 school years. Each of these schools experienced more than a 20-percentage-point 
gain in students attaining proficiency in both math and English language arts. In 2008, DC 
public schools will again be eligible for school performance awards based on changes in the 
percentages of students proficient at the school level. In later years, DC may switch to a 
value-added model. 

As with the other partners, NLNS’ goals in DC are to reward high-achieving schools, 
identify promising practices, and share those practices with others. However, the specifics of 
the program differ with each partner. For instance, in addition to awarding schools based on 
the change in percent of students proficient (rather than by the greatest value added in 
students’ scores), the distribution and magnitudes of the awards in DC are unique. First, DC 
distributed awards to noninstructional staff, including librarians, guidance counselors, and 
support staff. Second, only Gold-Gain awards were made in DC; the incentive program in 
DC did not include Silver-Gain awards. Finally, while the principal and assistant principal 



15 The DC eligible schools did not include charter schools, correctional facilities, family centers, or special 
education centers. 
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awards ($10,000 and $9,000 respectively) were similar to those in Memphis, teachers at the 
award-winning DC schools each received $8,000, compared with $1,500 per teacher at gold- 
winning Memphis and charter schools. 



Evaluation 

In this section, we explain the basic methodologies that MPR will employ to evaluate 
the incentive components of NLNS’ EPIC program in DC. We begin by describing how we 
will conduct descriptive analyses of our baseline data. We then discuss the obstacles to 
estimating the impact of incentives and then discuss analyses we can do of awareness and 
perceptions of EPIC incentives. Finally, we explain issues related to the power analysis and 
present the sampling design for the principal and teacher surveys. 

Descriptive Analysis 

To paint a baseline picture of each of the partners involved with EPIC, a series of 
school, student, and teacher characteristics will be compared across different groupings of 
schools in DC. As discussed in Chapter I, each partner provided us with student 
demographics and other student-level characteristics. We also will obtain school and teacher 
characteristics from the NCES Common Core of Data and other publicly available sources 
where possible. 

The analysis of baseline characteristics will be used, in part, to identify pre-existing 
differences among schools that could account for trends in student achievement, survey- 
response patterns, and so on. The primary unit of analysis will be the school. Student and 
teacher-level characteristics will first be calculated at a school level and then averaged across 
schools. The descriptive analysis for DC will compare schools grouped in a variety of ways. 
Initially, comparisons will be made across such groupings as: 

• Award status (award vs. no award) 

• School level (elementary or secondary) 

• Value-added quartile (top quartiles, middle two quartiles, bottom quartile) 16 

To explore the data further, we also may compare differences by grouping schools 
based on student demographics, school performance or teacher characteristics. Additionally, 
we will conduct baseline comparisons of schools based on responses to the principal survey. 
For instance, we will compare schools by whether principals are aware of EPIC, and whether 
principals have an understanding of EPIC eligibility requirements. 



16 Although value-added scores are not used in identifying Gold-Gain schools in DC in 2007/2008, MPR 
still has estimated value added scores for schools in DC. 
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Table III. 1 presents a table shell for comparing schools across these different groupings 
using the student and school information in DC. 1 This table shell, which is similar to those 
presented in Chapters II and IV, reflects the specific student characteristics available in DC. 
We will use t tests to test for significant differences between the groups. 

Table 111.1. Table Shell for Examining DC Schools 

Group A Group B Groups C, D, etc. 

School Characteristics 

School type 

Percent elementary 

Percent secondary 

Percent elementary & secondary 

Percent NCLB AYP pass overall 

NCLB AYP status (e.g., “in need of 
improvement”) 

Baseline value-added quartile 
Top quartile of schools 
Middle two quartiles 
Bottom quartile of schools 

Student Characteristics (Averages) 

Enrollment 

Number of days attended 

Ethnicity 

Percent black 
Percent Hispanic 

Percent male 
Percent free lunch 
Percent limited English 
Percent special education 
Percent retained 
Number of days suspended 
Number of expulsions 

Teacher Characteristics (Averages) 

Staff size 

Student/teacher ratio 



17 Note that we do not intend to present all of the various tables suggested by the table shell in III.l; 
rather, we will examine these tables to identify potentially meaningful differences and present those differences 
in our reports. 
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Evaluation of Incentives 

To estimate the impact of incentives on students’ test scores, we must compare 
outcomes from eligible schools with ineligible schools. We hypothesize that if educators are 
aware of the potential awards, they will be more motivated to increase their students’ test 
scores and hence, on average, students’ test-scores gains would be greater in award-eligible 
schools than in ineligible schools. Since all DC public schools are eligible for EPIC awards, 
we cannot perform this type of evaluation for DC. 

Although estimating the impact of the incentives is not possible, we can still examine 
survey responses and site visit data to gain valuable insight into this component of the EPIC 
program. For incentives to have an impact, educators need to be aware of their existence, 
want to improve their performance because of the incentives, and believe that they can do 
so. Using information obtained from the spring 2008 principal survey, we can examine all of 
these issues and estimate the fractions of school staff that are aware of EPIC, want to 
improve performance because of EPIC awards, and believe that they can do so. 

The DC Public Schools conducts an annual survey of all staff in their buildings. In 
addition to the principal and AP survey conducted in DC in 2008, we will add three 
questions for teachers to that survey. These questions will focus on teachers’ awareness of 
EPIC, their perception of performance-based incentives, and the sources they use to identify 
best practices in teaching. If the response rates to this survey are reasonably high (ideally 80 
percent or higher) we will be able to use the results to better understand whether teachers 
are aware of the incentives, whether they have negative feelings toward incentive programs, 
and whether they are likely to use the internet to identify effective practices. 

Survey Sample 

For the survey, the sample frame for DC will include all noncharter public schools 
except correctional facilities, family centers and special education centers. We will stratify the 
sample by the level of the school (elementary, middle or high) and whether the school 
principal was trained by NLNS and we will randomly select 80 DC schools from this frame 
for our 2008 spring survey. One important consideration is the fact that DC recently 
announced a large number of schools will close at the end of the 2007/2008 school year. We 
will exclude these schools from our sample frame. 

This sample should give us sufficient power to estimate means with 90 percent 
confidence intervals of +/- 11 percentage points for binary outcomes and +/- 0.16 standard 
deviations for continuous outcomes. lx When comparing subgroups of schools with 40 
schools in each subgroup we will be able to detect differences as small as 31 percentage 
points for binary outcomes and .63 standard deviations for continuous outcomes. 



18 These are based on two-tailed tests with 80 percent power and 5 percent statistical significance. The 
binary outcome is assumed to have a mean of 0.5. 
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Chapter IV 



Evaluating EPIC in 
Charter Schools 



I n this chapter, we summarize the results for the first round of awards for schools in the 
Charter School Consortium. We also discuss two possible options for estimating the 
incentive impact of EPIC for charter schools and describe how we will sample schools 
for the principal and teacher surveys for charter schools. 



Charter School Consortium 2007 Awards 

In 2007, 97 charter schools in 17 states and the District of Columbia were eligible for an 
EPIC school performance award. These schools were recruited into the EPIC program by 
NLNS. To be eligible, schools had to sign a memorandum of understanding with NLNS and 
provide MPR with the data needed to estimate a value-added model. 19 NLNS announced the 
first awards for charter schools in March 2008. Schools were chosen for awards based on 
value-added measures of their performance during the 2006/2007 school year. Schools with 
the highest value-added scores in their category (elementary, middle, or high) received Gold- 
Gain awards, and the next-highest-ranking schools in value-added scores received Silver- 
Gain awards. Schools that served multiple grade levels (e.g., both elementary and middle) 
were eligible for awards in all grade levels but could receive only one award. NLNS granted a 
total of seven Gold-Gain awards, which included three elementary schools, three middle 
schools, and one high school. Fifteen schools (six elementary, six middle schools, and three 
high schools) received Silver-Gain awards. The award-winning charter schools are from nine 
states and the District of Columbia. In spring 2008, NLNS will begin to visit Gold-Gain 
charter schools to identify promising practices and gauge educators’ reaction to EPIC. 

NLNS plans to recruit additional charter schools to be eligible for school performance 
awards and Spotlight Teacher awards in 2008/2009. It is unclear at this time how many 
additional schools may be added to the 97-school consortium. 



19 Two additional charter schools were close to being eligible but did not provide data on enough students 
to qualify for EPIC. 
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Evaluation 

In this section, we explain the methodologies that MPR will employ to evaluate the 
incentive components of NLNS’ EPIC program for charter schools. We begin by discussing 
the descriptive analysis we will conduct to gain insight into the baseline characteristics of the 
schools in the Charter School Consortium. Next, we review the concept of the difference -in- 
differences model, explain the difficulty in applying the model to the Charter School 
Consortium, and propose two options that may allow us to estimate the impact of the EPIC 
incentive using a difference-in-differences model. After that, we present a power analysis and 
describe how we are selecting schools for the principal and teacher charter school surveys. 

Descriptive Analysis 

The analysis of baseline characteristics will be used, in part, to identify pre-existing 
differences among schools that could account for trends in student achievement, survey- 
response patterns, and so on. The primary unit of analysis will be the school. Student and 
teacher-level characteristics will be calculated at a school level and then averaged across 
schools. The descriptive analysis for charter schools will compare schools grouped in a 
variety of ways. Initially, comparisons will be made across such groupings as: 

• Eligibility for EPIC 20 

• Award status (Gold-Gain vs. Silver-Gain vs. no award) 

• School level (elementary or secondary) 

• Value-added quartile (top quartiles, middle two quartiles, bottom quartile) 

• School characteristics (elementary or secondary, charter school authorizer type, 
age of school, whether part of CMO) 

• State 

• Treatment and comparison status 

To explore the data further, we also may compare differences by grouping schools 
based on student demographics, school performance, or teacher characteristics. Additionally, 
we will conduct baseline comparisons of schools based on responses to the principal survey. 
For instance, we will compare schools by whether principals are aware of EPIC and whether 
principals have an understanding of EPIC eligibility requirements. 



20 We will only have limited data on the EPIC ineligible schools but at a minimum should have value- 
added information, AYP status, and data available from the CCD. Those schools have not yet been identified. 
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Table IV. 1 presents a table shell for comparing schools across these different groupings 
using the student and school information for the charter schools." This table shell, which is 
similar to those presented in Chapters II and III, reflects the specific student characteristics 
available for the charter schools. We will use t tests to test for significant differences between 
the groups. 

Table IV.1. Table Shell for Examining Schools in the Charter School Consortium 

Group A Group B Groups C, D, etc. 



School Characteristics 

School type 

Percent elementary 

Percent secondary 

Percent elementary & secondary 

Percent NCLB AYP pass overall 

NCLB AYP status (e.g., “In need of improvement”) 

Baseline value-added quartile 
Top quartile of schools 
Middle two quartiles 
Bottom quartile of schools 

Charter school authorizer type 
Percent for-profit company 
Percent not-for-profit organization 
Percent school district/board 
Percent university 
Percent other 

Age of school 

Percent two years or younger 
Percent three to five years old 
Percent six years or older 

Student Characteristics (Averages) 

Enrollment 

Number of days enrolled 

Ethnicity 

Percent black 
Percent Hispanic 

Percent male 

Percent free or reduced price lunch 

Percent limited English 

Percent special education 

Teacher Characteristics (Averages) 

Staff size 

Student/teacher ratio 



21 Note that we do not intend to present all of the various tables suggested by the table shell in III.l; 
rather, we will examine these tables to identify potentially meaningful differences and present those differences 
in our reports. 
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Evaluation of Incentives 

To evaluate the incentive components of EPIC for charter schools, we propose 
estimating a difference-in-differences model similar to the model presented in Chapter II. 
The basic idea behind the model is to compare the gains in students’ scores between schools 
eligible for an award (the “treatment” schools) to the gains in students’ scores of ineligible 
schools (the “comparison” schools). If eligible schools have a greater incentive to improve 
students’ scores because of the potential awards, then we would expect the gain in students’ 
scores from eligible schools to be greater than the gain in students’ scores from ineligible 
schools when all else is equal. Although the methodology is straightforward, the difficulty 
arises in defining and/ or constructing the treatment and comparison schools for the Charter 
School Consortium. In Memphis, the district provided us with value-added data for eligible 
and ineligible schools. Thus, we could easily obtain a comparison group that was reasonably 
similar to the treatment group. For charter schools, we have value-added data only for the 
eligible charter schools. 

Below, we discuss two options for obtaining data on a comparison group of charter 
schools: (1) recmit non-eligible charter schools and (2) exploit differences in the degree of 
awareness of the EPIC program. 

Option 1 : RecruitingNon-Eligible Charter Schools 

Our first proposed strategy is to focus on recmiting charter schools from cities where 
we currently have at least five charter schools in the consortium: Chicago, DC, Los Angeles, 
and Oakland. This would allow us to estimate impacts of EPIC using our difference-in- 
differences model for each of these cities and then obtain a weighted average of the 
estimates across the cities. 

At present, we have data on DC Charter schools that are ineligible for the EPIC award; 
however, we would have to obtain data on other non-eligible charter schools. Since data 
collection can be difficult and time-consuming, it is unlikely that non-eligible charter schools 
would be willing to provide this data without compensation for their effort. To assess the 
willingness and the requirements of non-eligible charter schools to participate in this effort, 
we would have to contact potential charter schools. 

The approach we use will have implications for the minimum detectable difference. For 
instance, in 2007/2008 there were a total of 39 EPIC eligible charter schools from Chicago 
(16), DC (13), Los Angeles (5), and Oakland (5). If we use a matched-comparison approach 
with 39 treatment and 39 comparison schools, we could detect impacts on value-added 
measures as small as 0.11 standard deviations in terms of student test scores. The estimates 
would be more precise if we could obtain a larger sample of ineligible comparison charter 
schools. 

Option 2: Exploiting Differences in Degree of Awareness 

If we are unable to recruit a sufficient number of non-eligible charter schools to 
perform the type of analyses suggested in Option 1 , we may be able to exploit differences in 
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educators’ awareness of the EPIC program to examine the effect of the EPIC incentive for 
charter schools. 

Through conversations with NLNS, we understand that charter school principals and 
teachers within the consortium may differ considerably in their degree of awareness of the 
EPIC program. For many educators, incentive programs are controversial and believed to be 
unwelcome by teachers. It appears that in some cases, CMOs did not inform their principals 
of their participation in the program. In other situations, CMOs may have explained the 
program to their principals, but it is unclear if the principals communicated the information 
to their teachers. 

If we learn from answers to the 2008 principal survey that principal and/or teacher 
awareness varied noticeably among eligible charter schools, we may be able to exploit this 
variation and adapt our difference-in-differences model presented in Chapter II, substituting 
whether or not the principal and/or teachers in eligible charter schools were aware of their 
eligibility at time t, for T- the dummy variable indicating whether the school is in the 
“treatment” group. If exactly half of the schools made their staff aware of EPIC and half did 
not, we would have around 48 treatment schools and 48 comparison schools. This would 
enable us to detect impacts on value-added measures as small as 0.10 standard deviations in 
terms of student test scores. 

Survey Sample 

As described in Chapter I, the approach to examining the impact of effective practice 
dissemination will entail principal and teacher responses to questions about their practices. 
Unfortunately, since we have not yet identified any comparison schools, we cannot include 
them in our sampling frame. 

In Memphis and DC, we are conducting a survey of 80 schools. For the Charter School 
Consortium, we are increasing the sample size to 97 so we can cover all eligible charter 
schools. This is possible in part because the charter schools tend to be smaller and will, 
therefore, probably have fewer assistant principals. To explore the possibility of using 
variation in principal and teacher awareness to estimate the effect of the EPIC initiatives, our 
survey for charter schools will include questions focusing on when the principal and teachers 
became aware of the program. 
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New Leaders for New Schools 
School Administrator Survey 



List of Measures in New Leaders for New Schools EPIC Evaluation 

Question Measure 

A1 Number of hours in principal’s work week 

A2 Distribution of principal's work responsibilities 

A3 Principal participated in professional development 

Frequency, sources and content of professional development received by 
A4 principal 

Frequency of seeking information on best practices and rating of quality of 
A5 information by principal 

Frequency and types of student data used by principal to promote curriculum and 
A6 instruction 

A7 Frequency and types of student data used by principal to evaluate teachers 

B1 Importance and frequency of use of best practices by teachers 

Principal's perceptions of teachers' current ability, ability to improve teaching 
skills, commitment to students, belief about ability to move scores and 
B2 compensation relative to effort 

B3 Teachers participated in professional development 

B4 Frequency and content of professional development received by teachers 

B5 Characteristics of professional development received by teachers 

Principal's perceptions of current teacher compensation system and pay for 
Cl performance programs 

C2 Principal's awareness of the EPIC program 

C3 When principal first heard about EPIC 

C4 Where did principal first hear about EPIC 

C5 Principal's perception of school's eligibility for award 

C6 Principal's perception of likelihood of winning award and self-efficacy 

C7 Principal's knowledge about recipients and amounts of incentives 

Principal's knowledge about how the winning schools are selected, and 
C8 requirements to accept the award 

C9 Principal's perception of how participation in EPIC will affect school 

Cl 0 Principal's perception of teachers' awareness of EPIC 

Cl 1 Pay for performance program at school 

Cl 2 Other pay for performance program in district 

Principal's perception about level of support he receives from teachers, parents, 
D1-D2 and community 

E1-E10 School demographics 

F1-F10 Principal demographics 
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MATHEMATICA 

Policy Research, Inc. 



Effective Practices 
Evaluation 

Principal/Assistant Principal 

Survey 

Spring 2008 



Please complete survey: 



BY MAIL 
To: 

Kathy Sonnenfeld 
Survey Director 

Mathematica Policy Research, Inc. 
P.O. Box 2393 
Princeton, NJ 08543 



OR ON THE WEB 

Log on at: 

http://www.epe2008.org 

and enter your User ID and Password 



DEAR PRINCIPAL OR ASSISTANT PRINCIPAL: 



Your participation is important. Below are answers to some general questions. 

WHAT IS THE PURPOSE OF THIS SURVEY? 

The purpose of this survey is to obtain information about principals or assistant principals, such 
as principals’ experience and training, instructional priorities and interactions with the teachers 
and community. The survey will help us understand the existing teaching practices, school 
leadership approaches and school climate. 

WHO IS CONDUCTING THIS SURVEY? 

[The District of Columbia Public Schools/The Memphis City Schools/A consortium of charter 
schools that includes your school has partnered with New Leaders for New Schools (NLNS) to 
implement the Effective Practice Incentive Community (EPIC) initiative. Mathematica Policy 
Research, Inc., an independent research firm, is conducting the survey as part of an evaluation 
of the EPIC initiative. The initiative and evaluation are funded by a grant from the 
U.S. Department of Education. 

WHY SHOULD YOU PARTICIPATE IN THIS SURVEY? 

EPIC is a new initiative designed to help increase teacher effectiveness and student success. 
We will use the information collected to analyze the impact of the EPIC program. It should take 
you about 30 minutes to complete. In appreciation of your effort, we will provide a payment of 
$25.00. 

WILL YOUR RESPONSES BE KEPT CONFIDENTIAL? 

The information you provide through this survey will be kept strictly confidential. Responses will 
not be identified by individual or by school. 

WHO SHOULD YOU CONTACT WITH QUESTIONS? 

Please contact Kathy Sonnenfeld, Survey Director for this Effective Practices Evaluation, at 
mailto:ksonnenfeld@mathematica.com or call toll-free at 800-XXX-XXXX. 

HOW CAN YOU RESPOND TO THIS SURVEY? 

You can complete the survey on the web. Please go to the web address 
http://www.epe2008.org. To access the survey, you will need your unique user ID and 
password which are provided in the accompanying letter. If you do not have your unique user 
ID and password, please call toll-free at 800-XXX-XXX. 

OR 

You may complete this paper survey and return your completed survey in the enclosed, 
pre-addressed, postage-paid envelope to: 

Mathematica Policy Research, Inc. 

P.O. Box 2393 
Princeton, NJ 08543 
Attn: Kathy Sonnenfeld 




A. ADMINISTRATIVE PRACTICES AND PROFESSIONAL DEVELOPMENT 



A1 . On average, how many total hours do you work per week? 

| | | TOTAL HOURS WORKED PER WEEK 

A2. Thinking about the total number of hours you work per week, please indicate how these hours are divided 
among the following activities and whether you would like to spend more, less or about the same amount 
of time on each activity. 

In Column A, please estimate on average, how many hours per week you spend on each activity. 

In Column B, please Indicate if you would like to spend more, the same or less hours on each activity. 







COLUMN A 


COLUMN B 






Ala. Approximately how 
many hours per 
week do you spend 


Alb. Would you like to spend 
MORE, the SAME, or 
LESS time on... 






on... 


More 


Same 


Less 


a. 


Leadership (including defining and implementing school 
vision and culture, and working with school staff 
leadership teams)? 




i □ 


2 □ 


3 □ 


b. 


Student assessment and analysis of student data to 
inform curriculum and instruction 




i □ 


2 □ 


3 □ 


c. 


Classroom observations and providing teacher feedback 
from observations? 




i □ 


2 □ 


3 □ 


d. 


Curriculum and instructional activities (excluding 
classroom observations)? 




i □ 


2 □ 


3 □ 


e. 


Student management (including walking through school 
areas and student meetings/clubs)? 




i □ 


2 □ 


3 □ 


f. 


Working with the community (including parents, local 
district/board or Charter Management Organization 
(CMO))? 




, □ 


2 □ 


3 □ 


g- 


Your own and teacher professional development? 




i □ 


2 □ 


3 □ 


h. 


Management/administration (including human 
resources, budgeting, capital development, fund-raising, 
operations and logistics)? 




i □ 


2 □ 


3 □ 


i. Other (Please specify below) 


1 — 1 — 1 


i □ 


2 □ 


3 □ 


THE SUM OF ITEMS A- 1 SHOULD EQUAL THE TOTAL NUMBER OF 
HOURS WORKED PER WEEK AS LISTED IN Al. 


TOTAL 

HOURS | | | 
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A3. Thinking about the school year (2007-2008), did you spend any time on professional development? 

I i □ Yes 

y o □ No — > GO TO A5 



A4. These next questions are about the professional development you participated in during the school year 
(2007-2008). During the school year (2007-2008) . . . 

In Column A, for each area listed, please write in the number of days that you spent on professional development during the school year (2007- 
2008). If you did not spend any time on professional development in a specific area, please skip Column B for that area. 

In Column B, for each area listed, please mark the sources that provided the professional development that you participated in during the school 

year (2007-2008). 

T COLUMN! B 

COLUMN! A IWNRK (X) ALL THAT APPLY 



A4a. Approximately 
how many days 
did you spend 
on professional 
development in 
this area? 



A4b. Which sources provided the professional development that you participated 
in within this area? 







If you list zero (0) 
days, please skip 
COLUMN B for that 
area 
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a. 


Leadership 




1 □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


7 □ 


8 □ 


b. 


Student assessment 
methods 




1 □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


7 □ 


8 □ 


c. 


How to analyze and use 
student data to inform 
curriculum and instruction . 




1 □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


7 □ 


8 □ 


d. 


Literacy curriculum and 
instruction 




1 □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


7 □ 


8 □ 


e. 


Math curriculum and 
instruction 




1 □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


7 □ 


8 □ 


f. 


Science curriculum and 
instruction 




1 □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


7 □ 


8 □ 


g- 


Other curriculum and 
instruction (excluding 
literacy, math and science) 




1 □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


7 □ 


8 □ 


h. 


Teacher personnel issues 
(including evaluation, 
professional development, 
hiring) 




1 □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


7 □ 


8 □ 


i. Student behavior 

management 




1 □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


7 □ 


8 □ 


j. District/board or CMO 
issues 




1 □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


7 □ 


8 □ 


k. 


Parent/Community issues . 




1 □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


7 □ 


8 □ 


1. Management (including 
fund raising, budgeting, 
facilities maintenance, 
scheduling, non-teacher 
personnel issues) 




1 □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


7 □ 


8 □ 


m. 


Other (Please specify) 


1 1 1 


1 □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


7 □ 


8 □ 
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A5. These questions are about where you may have found information on teacher best practices. 





A5a. In the past six months, how 
often did you find information 
on teacher best practices from 
each of these sources? Would 
you say it is never, rarely, 
sometimes or frequently? 


A5b. How useful was the information on 
teacher best practices available from 
each of these sources? Would you say it 

was not at all useful, a little useful, 
somewhat useful, very useful or you never 
found information from this source? 




MARK (X) ONE FOR EACH ITEM 


MARK (X) ONE FOR EACH ITEM 




Never 


Rarely 


Sometimes 


Frequently 


Not at all 
useful 


A little 
useful 


Somewhat 

useful 


Very useful 


Never 

found 

information 


a. Educational journals (hard 
copy and online) 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


b. Peers and colleagues 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


c. Professional association .... 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


d. Internet resources 
(excluding education 
journals) 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


e. College/university courses 
taken as part of 
degree/certificate 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


f. College/university courses 
taken on your own 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


g. Conferences and other 
professional development .. 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 



The next set of questions is about how you use data in your school. 



A6. How often do you use each of the following data sources in your efforts to promote curriculum and 
instructional improvement"? Do you never, rarely, sometimes, frequently, or always use . . . 

MARK (X) ONE FOR EACH ITEM 





Never 


Rarely 


Sometimes 


Frequently 


Always 


a. Standardized test scores? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


b. Letter grades or GPAs? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


c. Rubric-based scoring of student work? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


d. Tests developed by teachers and other informal assessments? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


e. “Walk throughs” (less than 1 0 minutes long)? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


f. Direct observations of classrooms (at least 10 minutes long)? .. 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 



A7. How often do you use each of the following data sources in your efforts to evaluate teacher performance"? 

Do you never, rarely, sometimes, frequently, or always use . . . 

MARK (X) ONE FOR EACH ITEM 





Never 


Rarely 


Sometimes 


Frequently 


Always 


a. Standardized test scores? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


b. Letter grades or GPAs? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


c. Rubric-based scoring of student work? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


d. Tests developed by teachers and other informal assessments? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


e. “Walk throughs” (less than 1 0 minutes long)? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


f. Direct observations of classrooms (at least 10 minutes long)? . 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 
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B. TEACHER PRACTICES AND PROFESSIONAL DEVELOPMENT 



B1 . Now, we would like to le arn about the teaching practices at your school. 

Bla. How important is it to you that teachers 

at your school use the following Bib. During the school year (2007-2008), how 

practices? Would you say it is not at all often do teachers at your school use 

important , a little important , somewhat the following practices? Would you say 

important or very important to you that never, rarely, sometimes, frequently, or 

teachers at your school. . . always ? 



MARK (X) ONE FOR EACH ITEM I MARK (X) ONE FOR EACH ITEM 





Not at all 
Important 


A little 
Important 


Somewhat 

Important 


Very 

Important 


Never 


Rarely 


Sometimes 


Frequently 


Always 


a. Use formative 
assessments where 
appropriate in their 
classroom? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


b. Assess individual student 
progress regularly? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


c. Analyze and use student 
data to identify low 
performing students? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


d. Analyze and use student 
data to reteach content? .. 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


e. Analyze and use student 
data to revise teaching 
methods? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


f. Analyze and use student 
data to help students own 
goals and assess own 
learning progress? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


g. Define and communicate 
achievement standards 
and assessment criteria to 
all students? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


h. Use multiple teaching 
methods to respond to 
individual student learning 
styles (e.g. visual, 
auditory)? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


i. Adjust lessons to engage 
all students, including high 
and low performing 
students, in their 
classroom? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


j. Connect lesson content 
with students’ prior 
knowledge, life 
experiences and 
interests? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 
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Bla. How important is it to you that teachers 
at your school use the following 
practices? Would you say it is not at all 
important, a little important, somewhat 
important or very important to you that 
teachers at your school. . . 


Bib. During the school year (2007-2008), how 
often do teachers at your school use 
the following practices? Would you say 
never, rarely, sometimes, frequently, or 
always? 




MARK (X) ONE FOR EACH ITEM 


MARK (X) ONE FOR EACH ITEM 




Not at all 
Important 


A little 
Important 


Somewhat 

Important 


Very 

Important 


Never 


Rarely 


Sometimes 


Frequently 


Always 


k. Plan curriculum and 
lessons to align with state 
assessment standards?.... 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


1. Define, communicate, and 
model expected behavior 
to students? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


m. Use research-based 
instructional strategies to 
improve their teaching? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


n. Share their expertise with 
new teachers in the 
school? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


o. Formally share and 
collaborate within the 
school on best practices 
through structured 
activities and meetings? ... 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


p. Informally share and 
collaborate within the 
school on best practices?. 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


q. Disseminate their best 
practices via multi-media 
forums (e.g. website, 
videos) within their 
district? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


r. Observe or collaborate 
with teaching professionals 
outside the school? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


s. Communicate students’ 
achievement standards, 
assessment criteria and 
progress to parents? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


t. Collaborate with parents to 
identify strategies to 
achieve student learning? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


u. Take advantage of 
community resources to 
increase student learning 
opportunities (e.g. 
internships, funding and 
resources for student 
programs)? 


i □ 


2 □ 


3 □ 


4 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 
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B2. What proportion of the teachers at your school is characterized by the following? Would you say it is less 
than a quarter, about half, about three quarters or more than three quarters of the teachers at your school. . . 



MARK (X) ONE FOR EACH ITEM 







Less than 
a quarter 


About half 


About 

three 

quarters 


More than 
three 
quarters 


a. 


Have the skills to produce meaningful student learning? 


i □ 


2 □ 


3 □ 


4 □ 


b. 


Have the skills to manage student behavior? 


i □ 


2 □ 


3 □ 


4 □ 


c. 


Can learn to become effective teachers? 


i □ 


2 □ 


3 □ 


4 □ 


d. 


Are continually seeking new ideas to improve their teaching 
skills? 


, □ 


2 □ 


3 □ 


4 □ 


e. 


Work together with other teachers to do what is “best for 
students”? 


i □ 


2 □ 


3 □ 


4 □ 


f. 


Work hard to help their students succeed? 


, □ 


2 □ 


3 □ 


4 □ 


g- 


Believe that all students can succeed? 


i □ 


2 □ 


3 □ 


4 □ 


h. 


Take responsibility for improving the school? 


, □ 


2 □ 


3 □ 


4 □ 


i. 


Believe students’ success depend on factors outside of their 
control? 


i □ 


2 □ 


3 □ 


4 □ 


j- 


Believe best practices can improve student achievement?.... 


, □ 


2 □ 


3 □ 


4 □ 


k. 


Are overpaid for the amount of effort they put into their 
work? 


i □ 


2 □ 


3 □ 


4 □ 


1. 


Are underpaid for the amount of effort they put into their 
work? 


, □ 


2 □ 


3 □ 


4 □ 



B3. Did teachers at your school participate in professional development during this school year (2007-2008)? 

i □ Yes 

0 □ No — > GO TO Cl (PAGE 8) 
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B4. Thinking about the school year (2007-2008), please estimate how much time teachers at your school have 
spent on professional development in the areas listed below. Would you say teachers at your school spent no 
time, less than a day, one full day, 2-5 days, 6-10 days, or more than 10 days, on . . 



MARK (X) ONE FOR EACH ITEM 





No Time 


Less 
Than a 
Day 


One Full 
Day 


2-5 Days 


6-10 

Days 


More 
than 
10 days 


a. 


Methods to assess students? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


b. 


Methods to analyze and use student data 
to inform curriculum and instruction? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


c. 


Literacy curriculum and instruction? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


d. 


Math curriculum and instruction? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


e. 


Science curriculum and instruction? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


f. 


Other curriculum and instruction excluding 
literacy, math and science? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


g- 


Student behavior management? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


h. 


Working with parents/community? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 



B5. Thinking about the school year (2007-2008), how often was the teachers’ professional development 

characterized by the following? Would you say that the teachers’ professional development was never, rarely, 
sometimes, frequently or always . . . 



MARK (X) ONE FOR EACH ITEM 







Never 


Rarely 


Sometimes 


Frequently 


Always 


a. 


Designed or chosen to support the school’s 
improvement goals? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


b. 


Designed or chosen to support the district’s or 
charter management organization’s improvement 
goals? 


, □ 


2 □ 


3 □ 


4 □ 


5 □ 


c. 


Designed or chosen to support the implementation 
of state or local standards? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


d. 


Evaluated for evidence of improvement in student 
achievement? 


, □ 


2 □ 


3 □ 


4 □ 


5 □ 


e. 


Considered part of teachers’ regular work? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


f. 


Planned by teachers in this school or district? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


g- 


Presented by teachers in this school or district? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


h. 


Accompanied by the resources that teachers need 
(e.g., time and materials) to make changes in the 
classroom? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


i. 


Designed to allow teachers opportunities to 
participate in a network or learning community with 
other teachers within your school? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


j- 


Designed to allow teachers opportunities to 
participate in a network or learning community with 
other teachers outside your school? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 



Prepared by Mathematica Policy Research, Inc. 



3 / 26/08 







C. AWARDING TEACHERS 



Cl. The following are some general statements regarding teacher effectiveness and teacher compensation. To 
what extent do you agree or disagree with each of the following statements? Please indicate if you strongly 
disagree, disagree, agree or strongly agree. 



MARK (X) ONE FOR EACH ITEM 





Strongly 

Disagree 


Disagree 


Agree 


Strongly 

Agree 


a. 


Teachers’ pay should be primarily based on their level of 
education and years of teaching experience 


i □ 


2 □ 


3 □ 


4 □ 


b. 


Teacher’s pay should be partially based on an end-of-year 
evaluation of their practices by the principal 


i □ 


2 □ 


3 □ 


4 □ 


c. 


The current teacher salary system is satisfactory 


i □ 


2 □ 


3 □ 


4 □ 


d. 


Teachers’ pay should be tied partly to whether or not their 
students meet grade level standards 


i □ 


2 □ 


3 □ 


4 □ 


e. 


Teachers’ pay should be tied partly to the increase in test 
scores of the current school year’s class over last school 
year’s class 


i □ 


2 □ 


3 □ 


4 □ 


f. 


Teachers’ pay should be tied partly to the increase in test 
scores of the current school year’s class between the end of 
last school year and end of current school year 


i □ 


2 □ 


3 □ 


4 □ 


g- 


Teachers’ pay should be tied partly to the increase in test 
scores of the current school year’s class between the 
beginning and end of current school year 


i □ 


2 □ 


3 □ 


4 □ 


h. 


Teacher’s pay should be tied partly to the increase in the 
percent of students who score proficient on the state tests in 
the current year’s class compared to last year’s class 


i □ 


2 □ 


3 □ 


4 □ 


i. Teachers who help produce professional development 
materials should receive financial compensation 


i □ 


2 □ 


3 □ 


4 □ 


j. Rewards should be based on test scores at the school level 
and given to all teachers, regardless of how well students 
performed in the individual classrooms 


i □ 


2 □ 


3 □ 


4 □ 


k. 


Awarding selected teachers for higher student performance 
leads to counterproductive competition between teachers .... 


i □ 


2 □ 


3 □ 


4 □ 


1. Programs that reward all teachers based on school-level 
performance increase collaboration among teachers 


i □ 


2 □ 


3 □ 


4 □ 


m. 


Teacher incentive awards should be based on both teacher 
and school level performance 


i □ 


2 □ 


3 □ 


4 □ 



C2. Before receiving this survey, have you ever read or heard . . . 



MARK (X) ONE FOR EACH ITEM 





Yes 


No 


a. The name Effective Practice Incentive Community (EPIC)? 


1 □ 0 □ 


b. About a program that made substantial incentive awards in late 2007/early 2008 
to school staff in your district/charter school consortium for their students’ test 
score performance? 


1 □ 0 □ 



IF YOU ANSWERED NO TO ALL OF THE QUESTIONS FOR C2, PLEASE SKIP TO Cl 1 ON PAGE 1 1 
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For the rest of this section, we will refer to this incentive program as “EPIC.” 

C3. When did you first learn about EPIC? Was it . . . 

MARK (X) ONE BOX ONLY 

1 □ a month ago 

2 □ about three months ago 

3 □ about six months ago 

4 □ about a year ago, or 

5 □ more than a year ago? 

C4. Where did you first learn about EPIC? 

MARK (X) ONE BOX ONLY 

1 □ New Leaders for New Schools 

2 □ School district /board staff or charter management organization staff 

3 □ Work colleague (e.g., teacher, school staff) 

4 □ A friend/relative 

5 □ Read about it in a newspaper 
5 □ On the internet 

7 □ On TV 
a □ On the radio 

99 □ Other (Please describe) 

C5. Is your school eligible during the next school year (2008-2009) for the EPIC incentive award? 

i □ Yes 
o □ No 

d □ Don’t Know 

C6. For these next two questions (C6 and C7), assume your school is eligible for the EPIC incentive award in 
the 2008-2009 school year. To what extent to you agree or disagree with the following statements? Please 
indicate if you strongly disagree, disagree, agree or strongly agree. 

MARK (X) ONE FOR EACH ITEM 





Strongly 

Disagree Disagree 


Agree 


Strongly 

Agree 


My school is likely to receive an EPIC incentive award 


i □ 


2 □ 


3 □ 


4 □ 


More hard work on my part will increase the likelihood of my 
school receiving an EPIC incentive award 


i □ 


2 □ 


3 □ 


4 □ 
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C7. What is the dollar value of the school based incentive award that these following individuals could receive 
for the 2008-2009 school year under the EPIC program? Would you say none, less than $1,000, $1,001 to 
$5,000, $5,001 to $10,000, $10,001 to $20,000, $20,001 to $30,000, or more than $30,000 for a .. . 



MARK (X) ONE FOR EACH ITEM 





None 


Less than 
$1,000 


$1,001 to 
$5,000 


$5,001 to 
$10,000 


$10,001 to 
$20,000 


$20,001 to 
$30,000 


More than 
$30,000 


Don’t 

Know 


a. Principal 


o □ 


i □ 


2 □ 


3 □ 


4 □ 


sD 


6 □ 


d □ 


b. Assistant Principal 


o □ 


i □ 


2 □ 


3 □ 


4 □ 


sD 


6 □ 


d □ 


c. Teacher 


0 □ 


i □ 


2 □ 


3 □ 


4 □ 


sD 


6 □ 


d □ 


d. Teacher aide 


0 □ 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


6 □ 


d □ 


e. Administrative staff 


0 □ 


i □ 


2 □ 


3 □ 


4 □ 


sD 


6 □ 


d □ 


f. Custodial staff 


0 □ 


i □ 


2 □ 


3 □ 


4 □ 


sD 


6 □ 


d □ 


g. Social workers (guidance 

counselors, therapists) 


0 □ 


i □ 


2 □ 


3 □ 


4 □ 


sD 


6 □ 


d □ 



C8. The following are statements about the different components (criteria for the award and requirements to 
accept the award) of the EPIC award initiative. Is it true that . . . 



MARK (X) ONE FOR EACH ITEM 







True 


False 


Don’t 

Know 


a. 


Schools will be chosen for the award based on the increase in student test scores of 
the current year’s classes over last year’s classes 


i □ 


0 □ 


d □ 


b. 


Schools will be chosen for the award based on the increase in student test scores of 
the current year’s class between the end of last year and end of current year 


i □ 


0 □ 


d □ 


c. 


Schools will be chosen for the award based on the increase in percent of students 
who score proficient on state tests in the current year’s class compared to last year’s 
class 


i □ 


0 □ 


d □ 


d. 


Only the schools with the highest student scores will be chosen to receive the award 


, □ 


0 □ 


d □ 


e. 


Selected schools must meet a specified level of students who qualify for free or 
reduced-price lunch 


, □ 


0 □ 


d □ 


f. 


In order to receive an award, schools must allow an external group to visit their 
classrooms to identify effective practices 


, □ 


0 □ 


d □ 


g- 


In order to receive an award, schools must report test scores 


i □ 


0 □ 


d □ 


h. 


In order to receive an award, teachers must agree to provide documentation on their 
teaching practices 


i □ 


0 □ 


d □ 


i. 


In order to receive the award, teachers must agree to share their teaching practices 
with other schools 


i □ 


0 □ 


d □ 


j- 


The principal has to agree to accept the teacher award 


, □ 


0 □ 


d □ 


k. 


The teachers have to vote to accept the teacher award 


i □ 


0 □ 


d □ 


1 . 


The principal cannot accept his award unless the teachers agree to accept the 
teacher award 


i □ 


0 □ 


d □ 


m. 


Teachers can receive one award based on the performance of the school and 
another award for their own performance 


i □ 


0 □ 


d □ 
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C9. The following are statements about how participating in EPIC may affect your school. To what extent do 
you agree or disagree with each of the following statements? Please indicate if you strongly disagree, 
disagree, agree or strongly agree with each statement. 

MARK (X) ONE FOR EACH ITEM 







Strongly 

Disagree 


Disagree 


Agree 


Strongly 

Agree 


a. 


EPIC will be successful in boosting teacher effectiveness 
at my school 


i □ 


2 □ 


3 □ 


4 □ 


b. 


EPIC will lead to teachers teaching to the test rather than 
promoting student learning 


i □ 


2 □ 


3 □ 


4 □ 


c. 


Teachers at my school are excited about EPIC 


i □ 


2 □ 


3 □ 


4 □ 


d. 


EPIC will increase collaboration between teachers at my 
school 


, □ 


2 □ 


3 □ 


4 □ 


e. 


EPIC will increase competition between teachers at my 
school 


i □ 


2 □ 


3 □ 


4 □ 


f. 


The requirements to receive an EPIC award are reasonable . 


, □ 


2 □ 


3 □ 


4 □ 


g- 


EPIC will strengthen principal-teacher relations at my school 


i □ 


2 □ 


3 □ 


4 □ 



CIO. How aware are the teachers at your school of the EPIC initiative? Is it not at all aware, a little aware, 
somewhat aware, very aware or you don’t know? 

MARK (X) ONE BOX ONLY 

1 □ Not at all aware 

2 □ A little aware 

3 □ Somewhat aware 

4 □ Very aware 
d □ Don’t Know 

Cl 1 . Does your school have any programs to reward teachers for their performance based on student test 
scores? 

i □ Yes 

0 □ No 

d □ Don’t Know 

Cl 2. Are there other programs, other than the EPIC program, in the district that award schools for their 
performance based on student test scores? 

1 □ Yes 
o □ No 

d □ Don’t Know 
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D. SCHOOL COMMUNITY 



Now we’re going to ask you some questions about your school. 

D1. To what extent do you feel respected by the following members of the school community? Do you strongly 
disagree, disagree, agree or strongly agree that you feel respected by .. . 

MARK (X) ONE FOR EACH ITEM 







Strongly 

Disagree 


Disagree 


Agree 


Strongly 

Agree 


Not 

Applicable 


a. 


The teachers at this school? 


i □ 


2 □ 


3 □ 


4 □ 




b. 


The students at this school? 


i □ 


2 □ 


3 □ 


4 □ 




c. 


Parents? 


i □ 


2 □ 


3 □ 


4 □ 




d. 


Community leaders? 


i □ 


2 □ 


3 □ 


4 □ 




e. 


Regional/Area/Central Office or Charter 
Management Organization administration?.... 


i □ 


2 □ 


3 □ 


4 □ 


na □ 


f. 


Peers and colleagues? 


i □ 


2 □ 


3 □ 


4 □ 


na □ 



D2. To what extent do you agree or disagree with the following statements about teachers at your school? 

Please indicate if you strongly disagree, disagree, agree and strongly agree. 

MARK (X) ONE FOR EACH ITEM 







Strongly 

Disagree 


Disagree 


Agree 


Strongly 

Agree 


a. 


1 trust the teachers at their word 


i □ 


2 □ 


3 □ 


4 □ 


b. 


1 have confidence in the expertise of the teachers 


i □ 


2 □ 


3 □ 


4 □ 


c. 


1 feel supported by the teachers to try new ideas 


i □ 


2 □ 


3 □ 


4 □ 


d. 


It’s okay for teachers to discuss feelings, worries, and 
frustrations with me 


, □ 


2 □ 


3 □ 


4 □ 


e. 


Teachers get along well with each other 


i □ 


2 □ 


3 □ 


4 □ 


f. 


Teachers understand and support my vision and goals for 
the school 


i □ 


2 □ 


3 □ 


4 □ 
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E. YOUR SCHOOL DEMOGRAPHICS 



El . Are you the principal or assistant principal at 
this school? 

r i □ Principal 

2 □ Assistant Principal — > GO TO FI, PAGE 14 
E2. Which grades are offered in this school? 



MARK (X) ALL THAT APPLY 



1 □ Pre-K 

2 □ Kindergarten 

3 □ 1st 

4 □ 2nd 

5 □ 3rd 

6 □ 4th 

7 □ 5th 

8 □ 6th 



9 □ 7th 

10 □ 8th 

11 □ 9th 

12 □ 10th 

13 □ 11th 
H □ 12th 

is □ Ungraded 



E3. How many months is your school in session? 

1 □ Year round (12 months) 

2 □ August/September through May/June 

3 □ Other (Please specify) 



E4. On average during the 2007-2008 school year, 
how many total students were enrolled in this 
school (for the grades marked above in item 
E2)? 

I 1,1 I I I NUMBER OF STUDENTS 



E5. On average, how much time do teachers in this 
school spend preparing for their classes a 
week? Would you say it was . . . 

1 □ Less than 60 minutes 

2 □ 1-2 hours 

3 □ 2-4 hours 

4 □ 4-6 hours 

5 □ 6-8 hours 

6 □ 8-10 hours, or 

7 □ more than 10 hours? 



E6. On average during this school year (2007-2008), 
how many TEACHERS held FULL-TIME positions 
at this school? 

If none, please enter “0. ” 

| | | | NUMBER OF FULL-TIME TEACHERS 

E7. On average during this school year (2007-2008), 
how many TEACHERS held PART-TIME 
positions at this school? 

If none, please enter “0. ” 

I I I I NUMBER OF PART-TIME TEACHERS 



E8. During the last two years, about how many 
teachers... 

NUMBER 

a. Have been newly hired in your 

school? I | | 



b. Have you hired of your own 
choice? 



c. Have you encouraged to leave? . 



d. Have left and who you encouraged 
to leave? 



e. Have left for other reasons? 



E9. At the end of the last school year (2006-2007), 
did this school make Adequate Yearly Progress 
(AYP) in literacy, math and science? 

Adequate Yearly Progress is your state's measure of 
yearly progress toward achieving state academic 
standards. 





Yes 


No 


Literacy? 


i □ 


o □ 


Math? 


i □ 


0 □ 


Science? 


i □ 


0 □ 



E10. Have there been any major changes at your 
school during the 2007-2008 year that might 
impact academic performance of students, such 
as changes in curriculum, tests, school control 
(public, charter, private), or organization 
(multiple schools within a school), etc.? 

i □ Yes [please briefly describe this change) 



o □ No 
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F. ABOUT YOU 



FI . Are you male or female? 



i □ Male 



□ Female 



F2. What is your year of birth? 



F7. PRIOR to this school year (2007-2008), how 

many years did you serve as the principal and/or 
assistant principal of THIS OR ANY OTHER 
school? 

Count part of a year as 1 year. If none, please enter 



YEAR 



F3. Are you of Hispanic or Latino origin? 

i □ Yes 
o □ No 



F4. What is your race? 

MARK (X) ONLY ONE 

1 □ White/Caucasian 

2 □ Black/African-American 

3 □ Asian 

4 □ Native Hawaiian or Other Pacific Islander 

5 □ American Indian or Alaska Native 



F5. What is the highest degree you have earned? 

MARK (X) ONLY ONE 



1 □ Do not have a degree 

2 □ Associate’s degree .J® 

3 □ Bachelor’s degree (BA, B.S., etc.) — 

'4 □ Master’s degree (MA, MAT, MBA, M.Ed., 

M.S., etc.) 

5 □ Educational specialist or professional diploma 

(at least one year beyond master’s level) 

6 □ Doctorate or first professional degree 

(Ph.D., Ed.D., M.D., L.L.B., J.D., D.DS.) 



F6. Do you have a master’s degree or higher in 
Education Administration? 

i □ Yes 



YEAR(S) as principal/assistant principal 
in this school or any other school 



F8. PRIOR to this school year (2007-2008), how 
many years did you serve as the 
principal/assistant principal of THIS school? 

Count part of a year as 1 year. If none, please enter 



| | | YEAR(S) as principal/assistant principal 

in this school 



F9. BEFORE you became a principal/assistant 
principal, how many years of elementary or 
secondary teacher experience did you have? 

Count part of a year as 1 year. If none, please enter 



| | | YEAR(S) of teaching before becoming a 

principal/assistant principal 



F10. What is your current ANNUAL salary for your 
position in this school before taxes and 
deductions? 

If your position includes multiple duties (e.g., you 
teach a class and serve as principal at this school), 
please include your entire salary before taxes and 
deductions. Please report in whole dollars. 



0 10 1 PER YEAR 



o □ No 
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F1 1 . Do you anticipate returning to this school next year (2008-2009)? 



rC 



i □ Yes - 
o □ No 

d □ Don’t Know 



GO TO FI 3 



FI 2. Why might you leave? 

MARK ALL THAT APPLY 



1 □ 

2 □ 

3 □ 

4 □ 

5 □ 

6 □ 

7 □ 

8 □ 
9 □ 



wil 

wil 

wil 

wil 

wil 

wil 

wil 

wil 



be retiring 

be principal of another school within this district 

be principal of another school outside of this district 

be taking a position (other than principal) within this district 

be taking a position (other than principal) outside this district 

be leaving the education field to pursue other employment opportunities 

be leaving to take care of family 

be forced to leave due to school closing 



want to continue with my education full-time (Please specify area of study below) 



99 □ Another reason 

FI 3. What date did you complete this form? 

I_l_l / l_l_l / l_2_l _Q_I _Q_I_I 

Month Day Year 

Please report month as a number, such as 01 for January, and 02 for February. 
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FI 4. There are many reasons why people choose to complete a survey on paper or on the web when both 
options are available. Could you tell us why you chose to answer this survey on paper instead of on the 
web? For each reason, please indicate if you chose to complete this survey on paper instead of on the 
web because . . . 



MARK (X) ONE FOR EACH ITEM 







Yes 


No 


a. 


You did not have access to a computer 


i □ 


o □ 


b. 


Computers were in use by others at the times you wanted to complete the survey.... 


, □ 


0 □ 


c. 


You started the survey, but experienced technical problems. For example the 
screen froze or it took too long to load the page(s) 


i □ 


0 □ 


d. 


The computer screen was too small to read the questions 


i □ 


0 □ 


e. 


You were unable to read the questions on the screen because of the color scheme 
on the computer 


i □ 


0 □ 


f. 


You chose to complete the paper version of the survey because it was readily 
available 


i □ 


0 □ 


g- 


Another reason (Please specify) 


i □ 


0 □ 



FI 5. Thank you for completing this important survey. Please provide us with the following information so we 
can send you a payment of $25.00. Also we might need to contact you if we have any questions about 
answers you provided on the survey. 

Please PRINT your name, the address where you would like your payment sent (home or school) and the 
best telephone number and the most convenient time to reach you. 



Your Name: 

School Name: 

Please provide school name if you want the check to be sent to your school address 

Street Address: 

City: State: Zip Code: 

Work Telephone: (|_|_|_|) - 1_|_|_| - 1_|_|_|_| 

Area Code 

Home Telephone: (| | | |) - 1 | | | - 1 | | | | 

Area Code 

Cell Telephone: (| | | |) - 1 | | | - 1 | | | | 

Area Code 

Email Address: 



THANK YOU VERY MUCH FOR COMPLETING THIS SURVEY. 
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